Normalization Techniques in Deep Learning

Lei Huang, Ph.D., is an Associate Professor at Beihang University. His current research interests include normalization techniques involving methods, theories, and applications in training deep neural networks (DNNs).  He also has wide interests in representation and optimization of deep learning theory and computer vision tasks.  Dr. Huang serves as a reviewer for top-tier conferences and journals in machine learning and computer vision.

• Introduction.- Motivation and Overview of Normalization in DNNs.- A General View of Normalizing Activations.- A Framework for Normalizing Activations as Functions.- Multi-Mode and Combinational Normalization.- BN for More Robust Estimation.- Normalizing Weights.- Normalizing Gradients.- Analysis of Normalization.- Normalization in Task-specific Applications.- Summary and Discussion.