Publication Date



Technical Report: UTEP-CS-21-61a


Neural networks -- specifically, deep neural networks -- are, at present, the most effective machine learning techniques. There are reasonable explanations of why deep neural networks work better than traditional "shallow" ones, but the question remains: why neural networks in the first place? why not networks consisting of non-linear functions from some other family of functions? In this paper, we provide a possible theoretical answer to this question: namely, we show that of all families with the smallest possible number of parameters, families corresponding to neurons are indeed optimal -- for all optimality criteria that satisfy some reasonable requirements: : namely, for all optimality criteria which are final and invariant with respect to coordinate changes, changes of measuring units, and similar linear transformations.

tr21-61.pdf (164 kB)
Original file