A large number of objectives have been proposed to train latent variable
generative models. We show that many of them are Lagrangian dual functions of
the same primal optimization problem...
The primal problem optimizes the mutual
information between latent and visible variables, subject to the constraints of
accurately modeling the data distribution and performing correct amortized
inference. By choosing to maximize or minimize mutual information, and choosing
different Lagrange multipliers, we obtain different objectives including
InfoGAN, ALI/BiGAN, ALICE, CycleGAN, beta-VAE, adversarial autoencoders, AVB,
AS-VAE and InfoVAE. Based on this observation, we provide an exhaustive
characterization of the statistical and computational trade-offs made by all
the training objectives in this class of Lagrangian duals. Next, we propose a
dual optimization method where we optimize model parameters as well as the
Lagrange multipliers. This method achieves Pareto optimal solutions in terms of
optimizing information and satisfying the constraints.