🧐Variational Autoencoder (VAE)

Maximize Likelihood and KL divergence

Maximum Likelihood = Minimize KL Divergence

Lower Bound

How to generate$P_{\theta}(x|z)$👍

VAE lower Bound of $\log P_{\theta}(x)$

ELBO

Maximizing the likelihood of the observed $x$:

• $P(\bold{z})$: a normal distribution

• $P(x|\bold{z}) = N(x; \mu(\bold{z}), \sigma(\bold{z}))$, $\mu(\bold{z}), \sigma(\bold{z})$ is unknown and going to be estimated.

Loss: $L = \sum_i\log P(x^{(i)})$

It is straightforward to figure out the following:

Then the lower bound $L_b$ could be derived as follows:

• $q(\bold{z}|x)$ is also a normal distribution, which is estimated by a neural network. $q(\bold{z}|x)=N(\bold{z}; \mu'(x), \sigma'(x))$. In other words, the mean and variance of $\bold{z}$ are given by two functions $\mu'(\cdot)$ and $\sigma'(\cdot)$, which will be estimated by the ouput of a neural network.

• maximizing this lower bound needs to minimize $KL[q(\bold{z}|x) || P(\bold{z})]$ and maximize the second term, which will be connected with neural networks.

Minimize the first term $KL[q(\bold{z}|x) || P(\bold{z})]$

Maximize the second term

VAE and GMM

Problem in VAE