Communications in Mathematical Sciences

Volume 17 (2019)

Number 5

Dedicated to the memory of Professor David Shen Ou Cai

A priori estimates of the population risk for two-layer neural networks

Pages: 1407 – 1425

DOI: https://dx.doi.org/10.4310/CMS.2019.v17.n5.a11

Authors

Weinan E (Department of Mathematics and Program in Applied and Computational Mathematics, Princeton University, Princeton, New Jersey, U.S.A.; and Beijing Institute of Big Data Research, Beijing, China)

Chao Ma (Program in Applied and Computational Mathematics, Princeton University, Princeton, New Jersey, U.S.A.)

Lei Wu (Program in Applied and Computational Mathematics, Princeton University, Princeton, New Jersey, U.S.A.)

Abstract

New estimates for the population risk are established for two-layer neural networks. These estimates are nearly optimal in the sense that the error rates scale in the same way as the Monte Carlo error rates. They are equally effective in the over-parametrized regime when the network size is much larger than the size of the dataset. These new estimates are a priori in nature in the sense that the bounds depend only on some norms of the underlying functions to be fitted, not the parameters in the model, in contrast with most existing results which are a posteriori in nature. Using these a priori estimates, we provide a perspective for understanding why two-layer neural networks perform better than the related kernel methods.

Keywords

two-layer neural network, Barron space, population risk, a priori estimate, Rademacher complexity

2010 Mathematics Subject Classification

41A46, 41A63, 62J02, 65D05

Full Text (PDF format)

Received 28 April 2019

Accepted 25 July 2019

Published 6 December 2019