Conditional Neural Progress (CNPs)

ICML 18 10-23-2020

Motivation

CNPs combine benefits of NNs and GPs:

• the flexibility of stochastic processes such as GPs

• structured as NNs and trained via Gradient Descent from data directly

Model

we have a function $f(x_i)=y_i$ with input $x_i$ and output $y_i$ .

$f()$ is drawn from $P$ , a distribution over functions.

Define two sets:

• Observations: $O = \{(x_i, y_i)\}$

• Targets: $T = \{x_j\}$

Our goal is: given some observations, we want to be able to make predictions at unseen target inputs at test time. Just like supervise learning.

The architecture of our model captures this task:

• $r_i$ are the representations of the pairs $\{(x_i, y_i)\}$

• $r$ is the overall representation obtained by summing all $r_i$

• $h_{\theta}$ and $g_{\theta}$ are NNs

• $\phi_i$ parametrizes the output distribution (either a Gaussian or a categorical distribution)

Key properties of the model:

• CNPs are conditional distributions over functions trained to model the empirical conditional distributions of functions $f\sim P$ .

• CNPs are permutation invariant in O and T.

• scalable, achieving a running time complexity of $O(m+n)$ for making m predictions with n observations.

Reference

• https://arxiv.org/pdf/1807.01613.pdf

• https://vimeo.com/312299226

• https://www.martagarnelo.com/projects

• https://github.com/deepmind/neural-processes