This was part of Bayesian Statistics and Statistical Learning

Geometry of Linear Neural Networks that are Equivariant /  Invariant under Permutation Groups

Kathlén Kohn, KTH Royal Institute of Technology

Wednesday, December 13, 2023

Abstract: The set of functions parameterized by a linear fully-connected neural network is a determinantal variety. We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. Examples of such group actions are translations or 90◦ rotations on images. For such equivariant or invariant subvarieties, we provide an explicit description of their dimension, their degree as well as their Euclidean distance degree, and their singularities. We fully characterize invariance for arbitrary permutation groups, and equivariance for cyclic groups. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks, such as a weight sharing property, and we prove that all invariant linear functions can be learned by linear autoencoders.