> For the complete documentation index, see [llms.txt](https://lichangbin.gitbook.io/generative-models/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lichangbin.gitbook.io/generative-models/variational-autoencoder-vae.md).

# Variational Autoencoder (VAE)

## Maximize Likelihood and KL divergence

<figure><img src="/files/68RJv1Oe83CrRBxprHqk" alt="" width="563"><figcaption></figcaption></figure>

<figure><img src="/files/p1ymljVpINZmnhyeNDXc" alt="" width="563"><figcaption><p>Maximum Likelihood = Minimize KL Divergence</p></figcaption></figure>

## Lower Bound

<figure><img src="/files/Lt3xPHFONNxpQGA7fUV3" alt="" width="375"><figcaption></figcaption></figure>

How to generate$$P\_{\theta}(x|z)$$:thumbsup:

<figure><img src="/files/ErGS1DZFHqZbggZW67qu" alt="" width="375"><figcaption></figcaption></figure>

### VAE lower Bound of $$\log P\_{\theta}(x)$$

<figure><img src="/files/pfU7ZysTpZhDh6ZwwYEp" alt="" width="563"><figcaption></figcaption></figure>

## ELBO

Maximizing the likelihood of the observed $$x$$:

<figure><img src="/files/9LZBZbTntVjnI29lJ2oJ" alt="" width="176"><figcaption></figcaption></figure>

* $$P(\bold{z})$$: a normal distribution
* $$P(x|\bold{z}) =  N(x; \mu(\bold{z}), \sigma(\bold{z}))$$, $$\mu(\bold{z}), \sigma(\bold{z})$$ is unknown and going to be estimated.

Loss: $$L = \sum\_i\log P(x^{(i)})$$

It is straightforward to figure out the following:

<figure><img src="/files/WzzyEinATDyZdWsEAgyR" alt="" width="375"><figcaption></figcaption></figure>

Then the lower bound $$L\_b$$ could be derived as follows:

<figure><img src="/files/yOQVz0JhZRBWbfJ0E3kE" alt="" width="375"><figcaption></figcaption></figure>

* $$q(\bold{z}|x)$$ is also a normal distribution, which is estimated by a neural network. $$q(\bold{z}|x)=N(\bold{z}; \mu'(x), \sigma'(x))$$. In other words, the mean and variance of $$\bold{z}$$ are given by two functions $$\mu'(\cdot)$$ and $$\sigma'(\cdot)$$, which will be estimated by the ouput of a neural network.
* maximizing this lower bound needs to minimize $$KL\[q(\bold{z}|x) || P(\bold{z})]$$ and maximize the second term, which will be connected with neural networks.

### Minimize the first term $$KL\[q(\bold{z}|x) || P(\bold{z})]$$

<figure><img src="/files/XHulMHH0bVXR6iVSpmZL" alt="" width="375"><figcaption></figcaption></figure>

### Maximize the second term

<figure><img src="/files/2hPn45aRiD1JHqPexN1l" alt="" width="375"><figcaption></figcaption></figure>

##

## VAE and GMM

## Problem in VAE