> For the complete documentation index, see [llms.txt](https://lichangbin.gitbook.io/paper_notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lichangbin.gitbook.io/paper_notes/oct/conditional-neural-progress.md).

# Conditional Neural Progress (CNPs)

ICML 18   10-23-2020

## Motivation

![](https://1687130946-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MEnQbUIupyAn8eMmrmG%2F-MK_bnQLQAgnrSb2mhON%2F-MKaKHaNr16uEApMPncN%2Fimage.png?alt=media\&token=cbd2f237-c39c-4785-ad40-90d813c62f2b)

CNPs combine benefits of NNs and GPs:

* the flexibility of stochastic processes such as GPs
* structured as NNs and trained via Gradient Descent from data directly

## Model

![](https://1687130946-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MEnQbUIupyAn8eMmrmG%2F-MK_bnQLQAgnrSb2mhON%2F-MKaHXgFo4bTz9Ftznpc%2Fimage.png?alt=media\&token=9bfa9c1d-bca8-4110-b15b-17735d7f7852)

we have a function $$f(x\_i)=y\_i$$ with input $$x\_i$$ and output $$y\_i$$ .

$$f()$$ is drawn from $$P$$ , a distribution over functions.

Define two sets:

* Observations: $$O = {(x\_i, y\_i)}$$ 
* Targets: $$T = {x\_j}$$ 

Our goal is: given some observations, we want to be able to make predictions at unseen target inputs at test time. Just like supervise learning.

The architecture of our model captures this task:

![](https://1687130946-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MEnQbUIupyAn8eMmrmG%2F-MK_bnQLQAgnrSb2mhON%2F-MKaIhXYoxS_eNbOEnsS%2Fimage.png?alt=media\&token=37aa6543-0715-4314-ae8d-ad1b50d3748f)

* $$r\_i$$ are the representations of the pairs $${(x\_i, y\_i)}$$ 
* $$r$$ is the overall representation obtained by summing all $$r\_i$$ 
* $$h\_{\theta}$$ and $$g\_{\theta}$$ are NNs
* $$\phi\_i$$ parametrizes the output distribution (either a Gaussian or a categorical distribution)

#### Key properties of the model:

* CNPs are conditional distributions over functions trained to model the empirical conditional distributions of functions $$f\sim P$$ .
* CNPs are permutation invariant in O and T.
* scalable, achieving a running time complexity of $$O(m+n)$$ for making m predictions with n observations.

## Reference

* <https://arxiv.org/pdf/1807.01613.pdf>
* <https://vimeo.com/312299226>
* <https://www.martagarnelo.com/projects>
* <https://github.com/deepmind/neural-processes>