Communications in Information and Systems

Volume 20 (2020)

Number 3

Mathematical Engineering: A special issue at the occasion of the 85th birthday of Prof. Thomas Kailath

Guest Editors: Ali H. Sayed, Helmut Bölcskei, Patrick Dewilde, Vwani Roychowdhury, and Stephen Shing-Toung Yau

Second-order guarantees in centralized, federated and decentralized nonconvex optimization

Pages: 353 – 388

DOI: https://dx.doi.org/10.4310/CIS.2020.v20.n3.a4

Authors

Stefan Vlaski (Institute of Electrical Engineering, École Polytechnique Fédérale de Lausanne, Switzerland)

Ali H. Sayed (Institute of Electrical Engineering, École Polytechnique Fédérale de Lausanne, Switzerland)

Abstract

Rapid advances in data collection and processing capabilities have allowed for the use of increasingly complex models that give rise to nonconvex optimization problems. These formulations, however, can be arbitrarily difficult to solve in general, in the sense that even simply verifying that a given point is a local minimum can be NPhard [1]. Still, some relatively simple algorithms have been shown to lead to surprisingly good empirical results in many contexts of interest. Perhaps the most prominent example is the success of the backpropagation algorithm for training neural networks. Several recent works have pursued rigorous analytical justification for this phenomenon by studying the structure of the nonconvex optimization problems and establishing that simple algorithms, such as gradient descent and its variations, perform well in converging towards local minima and avoiding saddle-points. A key insight in these analyses is that gradient perturbations play a critical role in allowing local descent algorithms to efficiently distinguish desirable from undesirable stationary points and escape from the latter. In this article, we cover recent results on second-order guarantees for stochastic first-order optimization algorithms in centralized, federated, and decentralized architectures.

Received 15 March 2020

Published 2 December 2020