Variational Inference for Bayesian Neural Networks via Resolution of Singularities  [slides]

Abstract

In contrast to regular models, singular statistical models lack identifiability and a positive definite Fisher information matrix. Singular models are in fact ubiquitous in modern machine learning, with neural networks serving as a prominent example. Coinciding with the resurgence of interest in Bayesian neural networks, this work sets out to characterize the approximation gap in variational inference for the posterior distribution over neural network weights. The result rests on a central insight from singular learning theory according to which the posterior distribution over the parameters of a singular model, following an algebraic-geometrical transformation known as a desingularization map, is asymptotically a mixture of so-called standard forms. We proceed to demonstrate that a generalized gamma mean-field family, following desingularization, can recover the leading order term of the model evidence. The theoretical results are accompanied by a set of experiments in which we employ affine coupling layers to learn the unknown desingularization map.