1
$\begingroup$

I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me.

In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial a_j}\dfrac{\partial a_j}{\partial w_{ji}}$ (equation (5.50)), a notation is defined as $\delta_j\equiv\dfrac{\partial E_n}{\partial a_j}$ (equation (5.51)). In my understanding $\delta_j$ is the first part in the chain rule. However in equation (5.54), the book mentioned

As we have seen already, for the output units, we have $$\delta_k=y_k-t_k$$

Question 1: $y_k-t_k$ is the error on output unit $k$, which is simply the difference between the $k$th output unit value and the corresponding target value. But, from the definition of the notation $\delta_k$, we should have $$\delta_k=\dfrac{\partial \frac{1}{2}(y_k-t_k)^2}{\partial a_k}=(y_k-t_k)\dfrac{\partial y_k}{\partial a_k}=(y_k-t_k)\dfrac{\partial h(a_k)}{\partial a_k}$$ where $h(a_k)$ is the activation function. So why in the book $\delta_k=y_k-t_k$??

In page 242, Section 5.3. Error Backpropagation,

Consider a simple linear model where the outputs $y_k$ are linear combinations of the input variables $x_i$ so that $y_k=\sum_iw_{ki}x_i$. For a particular input pattern $n$, the error function is $E_n=\dfrac{1}{2}\sum_k(y_{nk}-t_{nk})^2$, where $y_{nk}=y_k(\boldsymbol{x_n},\boldsymbol{w})$. So the gradient of this error function with respect to a weight $w_{ij}$ is given by $$\frac{\partial E_n}{\partial w_{ji}}=(y_{nj}-t_{nj})x_{ni}$$ which can be interpreted as a ‘local’ computation involving the product of an ‘error signal’ $y_{nj} − t_{nj}$ associated with the output end of the link $w_{ji}$ and the variable $x_{ni}$ associated with the input end of the link.

Question 2: I am not clear with the structure of this neural network. The one in the book is a two-layer neural network with linear activation, is it?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

As you point out, with $\delta_k = y_k - t_k$, the author is stating the relationship between the final units' output and the target. So Equation 5.54 is simply stating:

the error on the $k^{th}$ output unit is the difference between its output and the target.

I believe that $\delta_k$ refers to the error simply at the output, whereas $\delta_j$ is the derivative of the output at unit $n$ with respect to any neuron back in the network, $j$. If this is the case, doing your derivative as you did, for $\delta_k$ (instead of $\delta_j$) means you are computing the error gradient between the output and the final layer. Over this layer, the activation must be linear (we do not apply a non-linearity at the final layers output). This would mean your $\dfrac{\partial h(a_k)}{\partial a_k}$ term would actually fall away in this case (to a constant), using the $k$ subscript - and you have the answer from the author: $y_t - t_k$.

The network that is considered in your second question seems to only really discuss a single fully-connected layer, so yes, two actual layers of neurons - input and output only with no activation (so a linear activation, as you mention).

Improve this answer
$\endgroup$
2
  • $\begingroup$ By "we do not apply a non-linearity at the final layers output", do you mean it is a common practice, or do you have a source for that statement? I'm new to neural network. $\endgroup$ Commented Aug 1, 2018 at 4:38
  • 1
    $\begingroup$ The final layer produces the prediction(s). In regression you'd expect a number. It'd be unnecessary to put that through a final non-linear activation function. In classification the final layer's output logits don't go through a non-linearity in order to increase the power of the network, rather just through a softmax to allow the output to be interpreted as probabilities (softmax squashes values into [0, 1]). Here is a reference. Imagine using a ReLU on the final layer... it'd make it impossible for the network to predict negative values. $\endgroup$ Commented Aug 1, 2018 at 11:45

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.