> For the complete documentation index, see [llms.txt](https://lichangbin.gitbook.io/paper_notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lichangbin.gitbook.io/paper_notes/meta-learning-of-neural-architectures-for-few-shot-learning.md). # Meta-Learning of Neural Architectures for Few-Shot Learning ## Motivation Few-shot learning is typically done with a fixed neural architecture. This paper proposes MetaNAS, the first method which fully integrates NAS with gradient-based meta learning. MetaNAS allows adapting architectures to novel tasks based on few data points with just a few steps of a gradient-based task optimizer. This allows MetaNAS to generate **task-specific architectures** that are adapted to every task separately (but from a joint meta- learned meta-architecture). ## Marrying Gradient-based Meta Learning and Gradient-based NAS * $$\alpha\_{meta}$$ : meta-learned architecture * $$w\_{meta}$$ : corresponding meta-learned weights for the architecture * Task $$\mathcal{T}\_i$$ : $$(\mathcal{D\_i^{tr}}, \mathcal{D\_i^{test}})$$ Meta-objective: $$ \begin{aligned} & \min\_{\alpha, w}\mathcal{L}*{meta}(\alpha, w, p^{train}, \Phi^k) \\ \= & \min*{\alpha, w}\sum\_{\mathcal{T}\_i \sim p^{train} }\mathcal{L}\_i\left(\Phi^k(\alpha, w, \mathcal{D}*i^{tr}), \mathcal{D}*i^{test}\right) \\ \= & \min*{\alpha, w}\sum*{\mathcal{T}\_i \sim p^{train} }\mathcal{L}*i\left((\alpha*{\mathcal{T}*i}^{\*},w*{\mathcal{T}\_i}^{\*}), \mathcal{D}\_i^{test}\right) \end{aligned} $$ where $$\alpha\_{\mathcal{T}*i}^{\*},w*{\mathcal{T}\_i}^{\*} = \Phi^k(\alpha, w, \mathcal{D}*i^{tr})=\operatorname{argmin}*{\alpha, w} \hat{\mathcal{L}}\_i (\alpha, w, \mathcal{D}\_i^{tr})$$ are the task-specific architecture and parameters after $$k$$ gradient steps, which could approximated using SGD. * $$\mathcal{L}\_i$$ : query loss for task $$i$$ * $$\hat{\mathcal{L}\_i}$$ : support loss for task $$i$$ Inner loop update $$\alpha, \text{and} \ \ w$$ with weight learning rate $$\xi\_{task}$$ and architecture learning rate $$\lambda\_{task}$$ : $$ \begin{aligned} \left(\begin{array}{c} \alpha^{j+1} \\ w^{j+1} \end{array}\right) &=\Phi\left(\alpha^{j}, w^{j}, D\_{i}^{tr}\right) \\ &=\left(\begin{array}{c} \alpha^{j}-\xi\_{\text {task}} \nabla\_{\alpha} \mathcal{L}*{\mathcal{T}}\left( \alpha^{j}, w^{j}, D*{i}^{tr}\right) \\ w^{j}-\lambda\_{\text {task}} \nabla\_{w} \mathcal{L}*{\mathcal{T}}\left(\alpha^{j}, w^{j}, D*{i}^{tr}\right) \end{array}\right) \end{aligned} $$ Outer loop update: $$ \begin{aligned} \left(\begin{array}{l} \alpha\_{\text {meta}}^{i+1} \\ w\_{\text {meta}}^{i+1} \end{array}\right) \=& \Psi^{M A M L}\left(\alpha\_{\text {meta}}^{i}, w\_{\text {meta}}^{i}, p^{\text {train}}, \Phi^{k}\right) \\ \= & \left(\begin{array}{c} \alpha\_{\text {meta}}^{i}-\xi\_{\text {meta}} \nabla\_{\alpha} \mathcal{L}*{\text {meta}}\left(\alpha*{\text {meta}}^{i}, w\_{\text {meta}}^{i}, p^{\text {train}}, \Phi^{k}\right) \\ w\_{\text {meta}}^{i}-\lambda\_{\text {meta}} \nabla\_{w} \mathcal{L}*{\text {meta}}\left(\alpha*{\text {meta}}^{i}, w\_{\text {meta}}^{i}, p^{\text {train}}, \Phi^{k}\right) \end{array}\right) \end{aligned} $$ Reptile could as be used instead of MAML here: $$ \begin{aligned} \left(\begin{array}{l} \alpha\_{\text {meta}}^{i+1} \\ w\_{\text {meta}}^{i+1} \end{array}\right) \=& \Psi^{Reptile}\left(\alpha\_{\text {meta}}^{i}, w\_{\text {meta}}^{i}, p^{\text {train}}, \Phi^{k}\right) \\ \= & \left(\begin{array}{c} \alpha\_{\text {meta}}^{i}+\xi\_{\text {meta}} \sum\_{\mathcal{T}*i}(\alpha*{\mathcal{T}*i}^{\*}-\alpha*{meta}^i)\\ w\_{\text {meta}}^{i} +\lambda\_{\text {meta}} \sum\_{\mathcal{T}*i^{\*}}(w*{\mathcal{T}*i}^{\*} - w*{meta}^{i}) \end{array}\right) \end{aligned} $$ ## Task-dependent Architecture Adaptation Two modifications to remove the need for retraining. ## Reference: * *