## Residual Normal Distribution

Introduced by Vahdat et al. in NVAE: A Deep Hierarchical Variational Autoencoder

Residual Normal Distributions are used to help the optimization of VAEs, preventing optimization from entering an unstable region. This can happen due to sharp gradients caused in situations where the encoder and decoder produce distributions far away from each other. The residual distribution parameterizes $q\left(\mathbf{z}|\mathbf{x}\right)$ relative to $p\left(\mathbf{z}\right)$. Let $p\left(z^{i}_{l}|\mathbf{z}_{<l}\right) := N \left(\mu_{i}\left(\mathbf{z}_{<l}\right), \sigma_{i}\left(\mathbf{z}_{<l}\right)\right)$ be a Normal distribution for the $i$th variable in $\mathbf{z}_{l}$ in prior. Define $q\left(z^{i}_{l}|\mathbf{z}_{<l}, x\right) := N\left(\mu_{i}\left(\mathbf{z}_{<l}\right) + \Delta\mu_{i}\left(\mathbf{z}_{<l}, x\right), \sigma_{i}\left(\mathbf{z}_{<l}\right) \cdot \Delta\sigma_{i}\left(\mathbf{z}_{<l}, x\right) \right)$, where $\Delta\mu_{i}\left(\mathbf{z}_{<l}, \mathbf{x}\right)$ and $\Delta\sigma_{i}\left(\mathbf{z}_{<l}, \mathbf{x}\right)$ are the relative location and scale of the approximate posterior with respect to the prior. With this parameterization, when the prior moves, the approximate posterior moves accordingly, if not changed.

#### Latest Papers

PAPER DATE
NVAE-GAN Based Approach for Unsupervised Time Series Anomaly Detection
Liang XuLiying ZhengWeijun LiZhenbo ChenWeishun SongYue DengYongzhe ChangJing XiaoBo Yuan
2021-01-08
NVAE: A Deep Hierarchical Variational Autoencoder
Arash VahdatJan Kautz
2020-07-08