We study the learning of finite mixture modeling in a singular setting,
where the number of model parameters is unknown. Simply overfitting
without suitable regularization of the model can result in highly inefficient
parameter estimation behavior.

To achieve the optimal rate of parameter estimation, we propose a
general framework for estimating the mixing measure arising in
finite mixture models, which we term minimum $Phi$-distance estimators.  
We establish a general theory for the general minimum $Phi$-distance
estimator, which involves obtaining sharp probability bounds on the
estimation error for the mixing measure in terms of the suprema of
the associated empirical processes for a suitably chosen function class
$Phi$. The theory makes clear the dual roles played by the function
class $Phi$, rich enough to induce necessary strong identifiability
conditions and small enough to produce optimal rates for parameter
estimation.

Our framework not only includes many existing estimation methods as special
cases but also results in new ones. For instance, it includes the minimum
Kolmogorov-Smirnov distance estimator as a special case, but also extends
to the multivariate setting.  It includes the method of moments as well,
while extending existing results for Gaussian mixtures to a larger family
of probability kernels. Moreover, the minimum $Phi$-distance estimation
framework leads to new methods applicable to complex (e.g., non-Euclidean)
observation domains.  In particular, we study a novel minimum distance
estimator based on the maximum mean discrepancy (MMD), a particular
$Phi$-distance that arises in a different context (of learning
with reproducing kernel Hilbert spaces).  

This work is joint with Yun Wei and Sayan Mukherjee.