Speaker: Dr. Qingguo Hong, Missouri University of Science and Technology | Department of Mathematics and Statistics
Date: Wednesday, September 20, 2023, 4:00 – 5:00 PM
Location: Science Engineering Hall (SCEN 408) / Zoom
Title: On the Activation Function Dependence of the Spectral Bias of Neural Networks
Abstract: Neural networks are universal function approximators which are known to generalize well despite being dramatically overparameterized. We study this phenomenon from the point of view of the spectral bias of neural networks. We provide a theoretical explanation for the spectral bias of ReLU neural networks by leveraging connections with the theory of finite element methods. Based upon this theory we predict that switching the activation function to a piecewise linear B-spline, namely the Hat function, will remove this spectral bias, which we verify empirically in a variety of settings. Our empirical studies also show that neural networks with the Hat activation function are trained significantly faster using stochastic gradient descent and ADAM. Combined with previous work showing that the Hat activation function also improves generalization accuracy on image classification tasks, this indicates that using the Hat activation provides significant advantages over the ReLU on certain problems.
Bio: Qingguo Hong, Ph.D. Dr. Hong is now holding an Assistant Professor position at the Department of Mathematics and Statistics in Missouri University of Science and Technology. Dr. Hong received his Ph.D. in Computational Mathematics from Peking University in 2012. After that, he joined Johann Radon Institute for Computational and Applied Mathematics (RICAM) at Austrian Academy of Sciences as a Research Scientist. He then moved to Faculty of Mathematics at The University of Duisburg-Essen in 2016 working as a Postdoctoral Scholar. Prior to joining Missouri University of Science and Technology, Dr. Hong worked as a Postdoctoral Scholar and an Assistant Research Professor at The Pennsylvania State University. Dr. Hong’s main research interest includes numerical analysis, numerical methods for PDEs, and machine learning. Dr. Hong has published papers on journals such as Math. Comp., SIAM J. Numer. Anal., Numer. Math., J. Comput. Physic., Comput. Methods Appl. Mech. Engrg., Math. Models Methods Appl. and so on.
Recent Comments