Variational Autoencoder (VAE)

PreviousOutline NextDiffusion Model

Last updated 10 months ago

Maximize Likelihood and KL divergence

Lower Bound

How to generate $P_{\theta}(x|z)$

VAE lower Bound of $\log P_{\theta}(x)$

ELBO

Maximizing the likelihood of the observed $x$ :

$P(\bold{z})$ : a normal distribution
$P(x|\bold{z}) = N(x; \mu(\bold{z}), \sigma(\bold{z}))$ , $\mu(\bold{z}), \sigma(\bold{z})$ is unknown and going to be estimated.

Loss: $L = \sum_i\log P(x^{(i)})$

It is straightforward to figure out the following:

Then the lower bound $L_b$ could be derived as follows:

$q(\bold{z}|x)$ is also a normal distribution, which is estimated by a neural network. $q(\bold{z}|x)=N(\bold{z}; \mu'(x), \sigma'(x))$ . In other words, the mean and variance of $\bold{z}$ are given by two functions $\mu'(\cdot)$ and $\sigma'(\cdot)$ , which will be estimated by the ouput of a neural network.
maximizing this lower bound needs to minimize $KL[q(\bold{z}|x) || P(\bold{z})]$ and maximize the second term, which will be connected with neural networks.

Minimize the first term $KL[q(\bold{z}|x) || P(\bold{z})]$

Maximize the second term

VAE and GMM

Problem in VAE

How to generate $P_{\theta}(x|z)$

VAE lower Bound of $\log P_{\theta}(x)$

Maximizing the likelihood of the observed $x$ :

$P(\bold{z})$ : a normal distribution

$P(x|\bold{z}) = N(x; \mu(\bold{z}), \sigma(\bold{z}))$ , $\mu(\bold{z}), \sigma(\bold{z})$ is unknown and going to be estimated.

Loss: $L = \sum_i\log P(x^{(i)})$

Then the lower bound $L_b$ could be derived as follows:

$q(\bold{z}|x)$ is also a normal distribution, which is estimated by a neural network. $q(\bold{z}|x)=N(\bold{z}; \mu'(x), \sigma'(x))$ . In other words, the mean and variance of $\bold{z}$ are given by two functions $\mu'(\cdot)$ and $\sigma'(\cdot)$ , which will be estimated by the ouput of a neural network.

maximizing this lower bound needs to minimize $KL[q(\bold{z}|x) || P(\bold{z})]$ and maximize the second term, which will be connected with neural networks.