Conditional Neural Progress (CNPs)

ICML 18 10-23-2020

Motivation

CNPs combine benefits of NNs and GPs:

we have a function $f(x_i)=y_i$ with input $x_i$ and output $y_i$ .

$f()$ is drawn from $P$ , a distribution over functions.

Define two sets:

Our goal is: given some observations, we want to be able to make predictions at unseen target inputs at test time. Just like supervise learning.

The architecture of our model captures this task:

$r_i$ are the representations of the pairs $\{(x_i, y_i)\}$
$r$ is the overall representation obtained by summing all $r_i$
$h_{\theta}$ and $g_{\theta}$ are NNs
$\phi_i$ parametrizes the output distribution (either a Gaussian or a categorical distribution)

CNPs are conditional distributions over functions trained to model the empirical conditional distributions of functions $f\sim P$ .
CNPs are permutation invariant in O and T.
scalable, achieving a running time complexity of $O(m+n)$ for making m predictions with n observations.

Last updated 4 years ago

Model

we have a function

f(x_i)=y_i

with input

x_i

and output

y_i

f()

is drawn from

P

, a distribution over functions.

Define two sets:

Observations: $O = \{(x_i, y_i)\}$

Targets: $T = \{x_j\}$

Our goal is: given some observations, we want to be able to make predictions at unseen target inputs at test time. Just like supervise learning.

The architecture of our model captures this task:

$r_i$ are the representations of the pairs $\{(x_i, y_i)\}$

$r$ is the overall representation obtained by summing all $r_i$

$h_{\theta}$ and $g_{\theta}$ are NNs

$\phi_i$ parametrizes the output distribution (either a Gaussian or a categorical distribution)

Key properties of the model:

CNPs are conditional distributions over functions trained to model the empirical conditional distributions of functions $f\sim P$ .

CNPs are permutation invariant in O and T.

scalable, achieving a running time complexity of $O(m+n)$ for making m predictions with n observations.