This was part of Permutation and Causal Inference

Inference for Synthetic Controls via Leave-Two-Out Placebo Tests

Lihua Lei, Stanford University

Tuesday, August 22, 2023

Abstract: The synthetic control method is often applied to problems with one or a few treated units and a small number of control units. Inference procedures that are justified asymptotically are often unsatisfactory due to (1) extremely small sample sizes that render large-sample approximation fragile and (2) unnecessary simplifications of the estimation procedure that is actually implemented in practice. A robust alternative is based on design-based inference (or conformal inference equivalently) which is closely related to the placebo test, a widely used diagnostic tool in practice. It provides valid Type-I error control in finite samples without artificial simplifications of the method when the treatment is assigned uniformly among units. Despite the robustness, it suffers from the low resolution since the null distribution is constructed from only N reference estimates, where N is the sample size. We propose a novel leave-two-out procedure that bypasses this issue by providing O(N^2) reference estimates while still maintaining the finite-sample Type-I error control under uniform assignments. Unlike the placebo test whose Type-I error always equals the theoretical upper bound, our procedure often achieves a much lower Type-I error than theory suggests and a higher power when the effect size is reasonably large. To account for deviation from uniform assignments, we generalize our procedure to allow for non-uniform assignments and show how to conduct sensitivity analysis based on quadratic programming.