This was part of Permutation and Causal Inference

Unifying Modes of Inference for Average Treatment Effects in Randomized Experiments

Colin Fogarty, University of Michigan

Thursday, August 24, 2023


Competing approaches to inference in randomized experiments differ primarily in (1) which notion of ``no treatment effect’’ being tested; and (2) whether or not stochasticity is assumed in the potential outcomes and covariates. Recommended hypothesis tests in a given paradigm may be invalid even asymptotically when applied in other frameworks, creating the risk of misinterpretation by practitioners when a given method is deployed. We develop a general framework for ensuring validity across competing modes of inference. We first describe a nested collection of bootstrap resampling schema providing valid inference of average treatment effects at differing levels of assumed stochasticity, ranging from superpopulation models, assuming random potential outcomes and covariates, to finite population inference, where only the assignment is viewed as random. To provide exact inference for stronger notions of no effect (such as Fisher's sharp null), we then employ permutation tests based upon prepivoted test statistics, wherein a test statistic is first transformed by a particular bootstrap cumulative distribution function and its permutation distribution is then enumerated. This provides a single mode of inference which is exact for sharp nulls, asymptotically valid for average treatment effects at the specified level of stochasticity, with higher order improvements for inference in superpopulation models.