Softmax vs Sigmoid function in Logistic classifier?

I've noticed people often get directed to this question when searching whether to use sigmoid vs softmax in neural networks. If you are one of those people building a neural network classifier, here is how to decide whether to apply sigmoid or softmax to the raw output values from your network:

• If you have a multi-label classification problem = there is more than one "right answer" = the outputs are NOT mutually exclusive, then use a sigmoid function on each raw output independently. The sigmoid will allow you to have high probability for all of your classes, some of them, or none of them. Example: classifying diseases in a chest x-ray image. The image might contain pneumonia, emphysema, and/or cancer, or none of those findings.
• If you have a multi-class classification problem = there is only one "right answer" = the outputs are mutually exclusive, then use a softmax function. The softmax will enforce that the sum of the probabilities of your output classes are equal to one, so in order to increase the probability of a particular class, your model must correspondingly decrease the probability of at least one of the other classes. Example: classifying images from the MNIST data set of handwritten digits. A single picture of a digit has only one true identity - the picture cannot be a 7 and an 8 at the same time.

Reference: for a more detailed explanation of when to use sigmoid vs. softmax in neural network design, including example calculations, please see this article: "Classification: Sigmoid vs. Softmax."

They are, in fact, equivalent, in the sense that one can be transformed into the other.

Suppose that your data is represented by a vector $\boldsymbol{x}$, of arbitrary dimension, and you built a binary classifier $P$ for it, using an affine transformation followed by a softmax:

\begin{equation} \begin{pmatrix} z_0 \\ z_1 \end{pmatrix} = \begin{pmatrix} \boldsymbol{w}_0^T \\ \boldsymbol{w}_1^T \end{pmatrix}\boldsymbol{x} + \begin{pmatrix} b_0 \\ b_1 \end{pmatrix}, \end{equation} \begin{equation} P(C_i | \boldsymbol{x}) = \text{softmax}(z_i)=\frac{e^{z_i}}{e^{z_0}+e^{z_1}}, \, \, i \in \{0,1\}. \end{equation}

Let's transform it into an equivalent binary classifier $P^*$ that uses a sigmoid instead of the softmax. First of all, we have to decide which is the probability that we want the sigmoid to output (which can be for class $C_0$ or $C_1$). This choice is absolutely arbitrary and so I choose class $C_1$. Then, my classifier will be of the form:

\begin{equation} z' = \boldsymbol{w}'^T \boldsymbol{x} + b', \end{equation} \begin{equation} P^*(C_1 | \boldsymbol{x}) = \sigma(z')=\frac{1}{1+e^{-z'}}, \end{equation} \begin{equation} P^*(C_0 | \boldsymbol{x}) = 1-\sigma(z'). \end{equation}

The classifiers are equivalent if the probabilities are the same for all $\boldsymbol{x}$, so we must impose:

\begin{equation} P^*(C_i|\boldsymbol{x})=P(C_i|\boldsymbol{x}) \quad i \in \{0,1\},\; \forall \boldsymbol{x}, \end{equation} or, equivalently, $\sigma(z') = \text{softmax}(z_1)$ for all $\boldsymbol{x}$. Now, replacing $z_0$, $z_1$, and $z'$ by their expressions in terms of $\boldsymbol{w}_0,\boldsymbol{w}_1, \boldsymbol{w}', b_0, b_1, b'$, and $\boldsymbol{x}$ and doing some straightforward algebraic manipulation, you may verify that the equality above holds if and only if $\boldsymbol{w}'$ and $b'$ are given by:

\begin{equation} \boldsymbol{w}' = \boldsymbol{w}_1-\boldsymbol{w}_0, \end{equation} \begin{equation} b' = b_1-b_0. \end{equation}

This shows that your first classifier $P$ (i.e., the one using the softmax) had more parameters than needed. This is true also for multiclass classification and it poses difficulties to optimization. An effective solution is to set the parameters for one of the classes to a fixed value (e.g., set $\boldsymbol{w}_0 = 0$ and $b_0=0$) and optimize only the remaining parameters.

Adding to all the previous answers - I would like to mention the fact that any multi-class classification problem can be reduced to multiple binary classification problems using "one-vs-all" method, i.e. having C sigmoids (when C is the number of classes) and interpreting every sigmoid to be the probability of being in that specific class or not, and taking the max probability.

So for example, in the MNIST digits example, you could either use a softmax, or ten sigmoids. In fact this is what Andrew Ng does in his Coursera ML course. You can check out here how Andrew Ng used 10 sigmoids for multiclass classification (adapted from Matlab to python by me), and here is my softmax adaptation in python.

Also, it's worth noting that while the functions are equivalent (for the purpose of multiclass classification) they differ a bit in their implementation (especially with regards to their derivatives, and how to represent y).

A big advantage of using multiple binary classifications (i.e. Sigmoids) over a single multiclass classification (i.e. Softmax) - is that if your softmax is too large (e.g. if you are using a one-hot word embedding of a dictionary size of 10K or more) - it can be inefficient to train it. What you can do instead is take a small part of your training-set and use it to train only a small part of your sigmoids. This is the main idea behind Negative Sampling.

[This is to show how the softmax arises naturally in the multiclass classification problem]

Ref: "Machine Learning Refined" by Watt, Borhani, Katsaggelos.

Consider data ${ \lbrace (\mathbf{x} _p, y _p) \rbrace _{p = 1} ^{P} }$ where each ${ \mathbf{x} _p \in \mathbb{R} ^N }$ and each ${ y _p \in \lbrace 0, 1, \ldots, C -1 \rbrace . }$

Let

$${ \overset{\circ}{\mathbf{x}} _p = (1, x _{1, p}, \ldots, x _{N, p}) ^T . }$$

Consider linear classifiers

$${ \mathbf{w} _0, \mathbf{w} _1, \ldots, \mathbf{w} _{C - 1} \in \mathbb{R} ^{N + 1} . }$$

Let

$${ b _j = w _{0, j}, \quad \boldsymbol{\omega} _j = (w _{1, j}, \ldots, w _{N, j}) . }$$

Consider the data point ${ (\mathbf{x} _p, y _p) . }$

Consider signed distances from the decision boundaries

$${ \frac{\mathbf{w} _j ^T \overset{\circ}{\mathbf{x}} _p}{\lVert \boldsymbol{\omega} _j \rVert} }$$

We want to pick ${ \mathbf{w} _0, \ldots, \mathbf{w} _{C - 1} }$ such that the label ${ y _p }$ can be predicted as the argmax of the above distances.

We want to pick ${ \mathbf{w} _0, \ldots, \mathbf{w} _{C -1} }$ such that

$${ \frac{\mathbf{w} _{y _p} ^T \overset{\circ}{\mathbf{x}} _p}{\lVert \boldsymbol{\omega} _{y _p} \rVert} \approx \max _{j = 0, \ldots, C - 1} \frac{\mathbf{w} _j ^T \overset{\circ}{\mathbf{x}} _p}{\lVert \boldsymbol{\omega} _j \rVert} \quad \text{ for } \, \, p = 1, \ldots, P . }$$

Hence consider the cost function

$${ {\begin{aligned} &\, g _{\text{init}} (\mathbf{w} _0, \ldots, \mathbf{w} _{C -1}) \\ = &\, \frac{1}{P} \sum _{p = 1} ^{P} \left[ \left( \max _{j = 0, \ldots, C - 1} \frac{\mathbf{w} _j ^T \overset{\circ}{\mathbf{x}} _p}{\lVert \boldsymbol{\omega} _j \rVert} \right) - \frac{\mathbf{w} _{y _p} ^T \overset{\circ}{\mathbf{x}} _p}{\lVert \boldsymbol{\omega} _{y _p} \rVert} \right] . \end{aligned}} }$$

Let

$${ \hat{\mathbf{w}} _j = \frac{\mathbf{w} _j}{\lVert \boldsymbol{\omega} _j \rVert} . }$$

Note that the cost function is

$${ {\begin{aligned} &\, g _{\text{init}} (\mathbf{w} _0, \ldots, \mathbf{w} _{C -1}) \\ = &\, \frac{1}{P} \sum _{p = 1} ^{P} \left[ \left( \max _{j = 0, \ldots, C - 1} \hat{\mathbf{w}} _j ^T \overset{\circ}{\mathbf{x}} _p \right) - \hat{\mathbf{w}} _{y _p} ^T \overset{\circ}{\mathbf{x}} _p \right] . \end{aligned}} }$$

Note that the max function can be smoothly approximated by the LogSumExp function. Link to Stackexchange post: Link.

Hence consider the cost function

$${ {\begin{aligned} &\, g (\mathbf{w} _0, \ldots, \mathbf{w} _{C -1}) \\ = &\, \frac{1}{P} \sum _{p = 1} ^{P} \left[ \log \left( \sum _{j = 0} ^{C - 1} \exp \left( \hat{\mathbf{w}} _j ^T \overset{\circ}{\mathbf{x}} _p \right) \right) - \hat{\mathbf{w}} _{y _p} ^T \overset{\circ}{\mathbf{x}} _p \right] \\ = &\, - \frac{1}{P} \sum _{p = 1} ^{P} \log \left(\frac{\exp \left( \hat{\mathbf{w}} _{y _p} ^T \overset{\circ}{\mathbf{x}} _p \right) }{\sum _{j = 0} ^{C - 1} \exp \left( \hat{\mathbf{w}} _j ^T \overset{\circ}{\mathbf{x}} _p \right)} \right) . \end{aligned}} }$$

Note that minimizing the cost can be viewed as maximizing the likelihood

$${ {\begin{aligned} &\, \prod _{p = 1} ^{P} {\color{blue}{p(y = y _p \vert \overset{\circ}{\mathbf{x}} _p, \mathbf{w} _0, \ldots, \mathbf{w} _{C-1})}} \\ = &\, \prod _{p = 1} ^{P} {\color{blue}{\frac{\exp \left( \hat{\mathbf{w}} _{y _p} ^T \overset{\circ}{\mathbf{x}} _p \right) }{\sum _{j = 0} ^{C - 1} \exp \left( \hat{\mathbf{w}} _j ^T \overset{\circ}{\mathbf{x}} _p \right)}}} . \end{aligned}} }$$

Hence we can consider the model generating the labels to be

$${ \boxed{ {\begin{aligned} &\, (y \vert \overset{\circ}{\mathbf{x}} , \mathbf{w} _0, \ldots, \mathbf{w} _{C-1}) \text{ is } \text{Categorical}(0, \ldots, K -1) , \\ &\, p(y = y _j \vert \overset{\circ}{\mathbf{x}} , \mathbf{w} _0, \ldots, \mathbf{w} _{C-1}) = \frac{\exp \left( \hat{\mathbf{w}} _{y _j} ^T \overset{\circ}{\mathbf{x}} \right) }{\sum _{j = 0} ^{C - 1} \exp \left( \hat{\mathbf{w}} _j ^T \overset{\circ}{\mathbf{x}} \right)} \end{aligned}} } }$$

and perform maximum likelihood estimation, as needed.