Why Rectified Linear Neurons Are Efficient: Symmetry-Based, Complexity-Based, and Fuzzy-Based Explanations

Authors

Olac Fuentes, The University of Texas at El PasoFollow
Justin Parra, The University of Texas at El PasoFollow
Elizabeth Y. Anthony, The University of Texas at El PasoFollow
Vladik Kreinovich, The University of Texas at El PasoFollow

Publication Date

12-2017

Comments

Technical Report: UTEP-CS-17-76a

To appear in: Olga Kosheleva, Sergey Shary, Gang Xiang, and Roman Zapatrin (eds.), Beyond Traditional Probabilistic Data Processing Techniques: Interval, Fuzzy, etc. Methods and Their Applications, Springer, Cham, Switzerland, 2018.

Abstract

Traditionally, neural networks used a sigmoid activation function. Recently, it turned out that piecewise linear activation functions are much more efficient -- especially in deep learning applications. However, so far, there have been no convincing theoretical explanation for this empirical efficiency. In this paper, we show that, by using different uncertainty techniques, we can come up with several explanations for the efficiency of piecewise linear neural networks. The existence of several different explanations makes us even more confident in our results -- and thus, in the efficiency of piecewise linear activation functions.