1
$\begingroup$

I was recently looking at the differences between an ANN (Artificial Neural Network) and a PNN (Probablistic Neural Network).

The PNN is described here: PNN

The ANN is described here: ANN

My question is this: there are several differences between the ANN and the PNN, and most of these differences seem like additional layers added onto the PNN (both being feed-forward nets of course). If you had already had an existing ANN that made empirical decisions (yes/no) for its output, would turning that output into a probability value from 0 to 1 be as simple as reading the raw output from the last neuron (a sigmoid function), or would this not be an accurate probability measurement?

Okay, here is the edit to my question:

Each of the circles below are artificial neurons that each have two parts: the first part is a weighted sum of the inputs, then that result is fed into the second part of the neuron which is the sigmoid (logistic) function (with a range from 0 to 1).

neural_net_pic

Now, on a typical training/testing demo, the network is trained first using back propagation, then the network is tested. When its tested, typically the network output $Y$ is used as a classifier only. Meaning that if the sigmoid output is $0 \le Y \lt 0.5$, then it positive (1). Otherwise if it is $0.5 \le Y \le 1$, then it is negatively classified (0).

My question was if we can do this, then during the testing phase, could we just read the raw output of the top neuron (which is a sigmoid output between 0 and 1) and treat it as a probability instead of a classification? If not, then why not?

$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

That is a great question. Largely the structure of the network is independent of the results it predicts of course, what matters is your data.

The final softmax output of any classification based artificial neural network is in fact a "probability", but it is not the probability of the data itself that you are trying to model, but rather the probability that the network believes the class is correct. Now, if your data is from a VERY uniform distribution, these should converge to the same thing. The PNN that page you linked describes is discussing the network structure I'm talking about, not an actual model for predicting probability.

If you want a network to predict the probability of an input event happening or not more accurately, you can modify your data in ways that are very specific to your problem. For example, you can have your event have an $n$ output neurons where $100/n$ is the confidence interval you want (10 classes would give a 10% probability threshold) and then your labels would be the observed labeled probability of the training events. You could also attempt to weight your samples with how good the input is an indicator of the output to have the final softmax value more equal the observed underlying probability. If you describe your problem in more detail I would be happy to help you come up with a more specific model.

EDIT:

The detail you added to the question does not change the answer. The natural tendency of back propagation through momentum based optimizers (or even SGD) is to have relatively large weights towards final layers push the input to the output softmax neuron to a very large or very small value effectively having a nearly 0 or nearly 1 prediction. You could, if your training data is very well distributed and you use a regularizer (like kernel/weight l2) then you can treat the direct softmax output as a probability. However, suppose your data is very separable in the networks dimensionality... as in the plane drawn to separate the points:

enter image description here

has a very large margin, then that probability will tend not to represent any ground truth closeness to the hyperplane, but instead will have sigmoid crush to 0 or 1. You can experimentally show this by using randomly sample NN input to show that the output probability does not follow a normal distribution.

In summary, yes you can in theory use the softmax output as a probability. This does however require extremely careful network structure, proper fitting, correct regularization, and good data. Most ANNs will decide to use the softmax in a less continuous way. Does this make more sense?

$\endgroup$
1
  • $\begingroup$ Nucl3ic, I just edited the question to explain a bit more in depth what I mean $\endgroup$ Commented Aug 1, 2017 at 19:18

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.