25
$\begingroup$

I am currently preparing for an exam on neural networks. In several protocols from former exams I read that the activation functions of neurons (in multilayer perceptrons) have to be monotonic.

I understand that activation functions should be differentiable, have a derivative which is not 0 on most points, and be non-linear. I do not understand why being monotonic is important/helpful.

I know the following activation functions and that they are monotonic:

  • ReLU
  • Sigmoid
  • Tanh
  • Softmax: I'm not sure if the definition of monotonicity is applicable for functions $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$ with $n, m > 1$
  • Softplus
  • (Identity)

However, I still can't see any reason why for example $\varphi(x) = x^2$.

Why do activation functions have to be monotonic?

(Related side question: is there any reason why the logarithm/exponential function is not used as an activation function?)

Improve this question
$\endgroup$
10
  • 4
    $\begingroup$ FYI: Comprehensive list of activation functions in neural networks with pros/cons $\endgroup$ Commented Dec 7, 2015 at 1:13
  • 2
    $\begingroup$ @MartinThoma Are you sure softmax is monotonic? $\endgroup$ Commented Feb 21, 2018 at 7:07
  • 2
    $\begingroup$ Thanks @Media. To answer your question: I'm not sure what "monotonic" even means for functions in $f:R^n \rightarrow R^m$ with $m > 1$. For $m=1$ softmax is constant and thus monotonic. But without defining $<$ for elements in $R^n$ with $n>1$ I don't think monotonic makes any sense. $\endgroup$ Commented Feb 21, 2018 at 19:50
  • 2
    $\begingroup$ @MartinThoma Thanks, actually it was also a question of mine. I didn't know, and still don't know, if there is an extension for monotonic in functions with multiple outputs. Math stuff, you know! $\endgroup$ Commented Feb 22, 2018 at 14:06
  • 1
    $\begingroup$ @GreenFalcon There is a not a unique way to do this, but there are ways. E.g. $\vec x < \vec y \implies f(\vec x) < f(\vec y)$ where $<$ is taken be the respective strict product orders on $\mathbb{R}^n$ and $\mathbb{R}^m$. $\endgroup$ Commented Sep 22, 2022 at 17:37

2 Answers 2

Reset to default
19
$\begingroup$

The monotonicity criterion helps the neural network to converge easier into an more accurate classifier. See this stackexchange answer and wikipedia article for further details and reasons.

However, the monotonicity criterion is not mandatory for an activation function - It is also possible to train neural nets with non-monotonic activation functions. It just gets harder to optimize the neural network. See Yoshua Bengio's answer.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Can you please elaborate what do you mean by "harder to optimize"? If you mean that using non-monotonic functions increases the number of local minima (hence decreasing the problem's convexity), is there any research pointing to this proof? $\endgroup$ Commented Dec 6, 2022 at 22:43
0
$\begingroup$

I will provide a more mathematical reason as to why does a having a monotone function helps!

Using http://mathonline.wikidot.com/lebesgue-s-theorem-for-the-differentiability-of-monotone-fun, assuming our activation function to be monotone, we can say that on the real line, our function will be differentiable. So, the gradient of the activation function will not be a erratic function. It will be easier to find the minima we are looking for. (computationally inexpensive)

Exponential and Logarithmic functions are beautiful functions but are not bounded(So, the converse of Lebesgue Theorem is not true as Exp and Log are differentiable functions which are not bounded on the real line). So, they fail when we want to classify our examples at the final stage. Sigmoid and tanh work really well because they have gradients which are easy to compute and their range is (0,1) and (-1,1) respectively.

Improve this answer
$\endgroup$
1
  • 6
    $\begingroup$ There are infinitely many differentiable, but not monotone functions. So why does having a monotone function help? $\endgroup$ Commented Aug 4, 2019 at 12:41

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.