Why neural networks in the first place: a theoretical explanation

Authors

Jonatan Contreras, The University of Texas at El PasoFollow
Martine Ceberio, The University of Texas at El PasoFollow
Olga Kosheleva, The University of Texas at El PasoFollow
Vladik Kreinovich, The University of Texas at El PasoFollow

Publication Date

10-1-2021

Comments

Technical Report: UTEP-CS-21-61a

Abstract

Neural networks -- specifically, deep neural networks -- are, at present, the most effective machine learning techniques. There are reasonable explanations of why deep neural networks work better than traditional "shallow" ones, but the question remains: why neural networks in the first place? why not networks consisting of non-linear functions from some other family of functions? In this paper, we provide a possible theoretical answer to this question: namely, we show that of all families with the smallest possible number of parameters, families corresponding to neurons are indeed optimal -- for all optimality criteria that satisfy some reasonable requirements: : namely, for all optimality criteria which are final and invariant with respect to coordinate changes, changes of measuring units, and similar linear transformations.