Why Rectified Linear Neurons: Two Convexity-Related Explanations

Authors

Jonatan Contreras, The University of Texas at El PasoFollow
Martine Ceberio, The University of Texas at El PasoFollow
Olga Kosheleva, The University of Texas at El PasoFollow
Vladik Kreinovich, The University of Texas at El PasoFollow
Nguyen Hoang Phuong, Thang Long UniversityFollow

Publication Date

6-1-2021

Comments

Technical Report: UTEP-CS-21-60

Abstract

At present, the most efficient machine learning technique is deep learning, in which non-linearity is attained by using rectified linear functions s(x)=max(0,x). Empirically, these functions work better than any other nonlinear functions that have been tried. In this paper, we provide a possible theoretical explanation for this empirical fact. This explanation is based on the fact that one of the main applications of neural networks is decision making, when we want to find an optimal solution. We show that the need to adequately deal with situations when the corresponding optimization problem is feasible -- i.e., for which the objective function is convex -- uniquely selects rectified linear activation functions.