Are Saddles Good Enough for Deep Learning?

Adepu Ravi Sankar Vineeth N Balasubramanian RS Adepu
Abstract: Recent years have seen a growing interest in understanding deep neural networks from an optimization perspective. It is understood now that converging to low-cost local minima is sufficient for such models to become effective in practice. However, in this work, we propose a new hypothesis based on recent theoretical findings and empirical studies that deep neural network models actually converge to saddle points with high degeneracy. Our findings from this work are new, and can have a significant impact on the ...