The excellent generalizability of deep learning is like a “cloud” to conventional complexity-based learning theory: the over-parameterization of deep learning makes almost all existing tools vacuous.
Introduction.- Background.- Conventional Statistical Learning Theory.- Difficulty of Conventional Statistical Learning Theory.- Developing Deep Learning Theory.- Generalization Bounds on Hypothesis Complexity.- Interplay of Optimization, Bayesian Inference, and Generalization.- Geometrical Properties of Loss Surface.- The Role of Over-parametrization.- Rising Concerns in Ethics and Security.- Privacy Preservation.- Fairness Protection.- Algorithmic Robustness.