# Variational Autoencoder (VAE)

## Maximize Likelihood and KL divergence

<figure><img src="/files/68RJv1Oe83CrRBxprHqk" alt="" width="563"><figcaption></figcaption></figure>

<figure><img src="/files/p1ymljVpINZmnhyeNDXc" alt="" width="563"><figcaption><p>Maximum Likelihood = Minimize KL Divergence</p></figcaption></figure>

## Lower Bound

<figure><img src="/files/Lt3xPHFONNxpQGA7fUV3" alt="" width="375"><figcaption></figcaption></figure>

How to generate$$P\_{\theta}(x|z)$$:thumbsup:

<figure><img src="/files/ErGS1DZFHqZbggZW67qu" alt="" width="375"><figcaption></figcaption></figure>

### VAE lower Bound of $$\log P\_{\theta}(x)$$

<figure><img src="/files/pfU7ZysTpZhDh6ZwwYEp" alt="" width="563"><figcaption></figcaption></figure>

## ELBO

Maximizing the likelihood of the observed $$x$$:

<figure><img src="/files/9LZBZbTntVjnI29lJ2oJ" alt="" width="176"><figcaption></figcaption></figure>

* $$P(\bold{z})$$: a normal distribution
* $$P(x|\bold{z}) =  N(x; \mu(\bold{z}), \sigma(\bold{z}))$$, $$\mu(\bold{z}), \sigma(\bold{z})$$ is unknown and going to be estimated.

Loss: $$L = \sum\_i\log P(x^{(i)})$$

It is straightforward to figure out the following:

<figure><img src="/files/WzzyEinATDyZdWsEAgyR" alt="" width="375"><figcaption></figcaption></figure>

Then the lower bound $$L\_b$$ could be derived as follows:

<figure><img src="/files/yOQVz0JhZRBWbfJ0E3kE" alt="" width="375"><figcaption></figcaption></figure>

* $$q(\bold{z}|x)$$ is also a normal distribution, which is estimated by a neural network. $$q(\bold{z}|x)=N(\bold{z}; \mu'(x), \sigma'(x))$$. In other words, the mean and variance of $$\bold{z}$$ are given by two functions $$\mu'(\cdot)$$ and $$\sigma'(\cdot)$$, which will be estimated by the ouput of a neural network.
* maximizing this lower bound needs to minimize $$KL\[q(\bold{z}|x) || P(\bold{z})]$$ and maximize the second term, which will be connected with neural networks.

### Minimize the first term $$KL\[q(\bold{z}|x) || P(\bold{z})]$$

<figure><img src="/files/XHulMHH0bVXR6iVSpmZL" alt="" width="375"><figcaption></figcaption></figure>

### Maximize the second term

<figure><img src="/files/2hPn45aRiD1JHqPexN1l" alt="" width="375"><figcaption></figcaption></figure>

##

## VAE and GMM

## Problem in VAE


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://lichangbin.gitbook.io/generative-models/variational-autoencoder-vae.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.