Error Calculation
Error calculation is an important aspect of any neural network. Whether the neural network is supervised or unsupervised, an error rate must be calculated. The goal of virtually all training algorithms is to minimize the error. In this section we will examine how the error is calculated for a supervised neural network. We will also discuss how the error is determined for an unsupervised training algorithm. We will begin this section by discussing two error calculation steps used for supervised training.
Error Calculation and Supervised Training
Error calculation is an important part of the supervised training algorithm. In this section we will examine an error calculation method that can be employed by supervised training. Further, we will see how this training algorithm is applied to the neural network class that was introduced in Chapter 3.
For supervised training there are two components to the error that must be considered. First, we must calculate the error for each of the training sets as they are processed. Secondly we must take the average across each sample for the training set. For example, the XOR problem that has only four items in its training set. An output error would be calculated on each element of the training set. Finally, after all training sets have been processed, the root mean square (RMS) error is determined.
Output Error
The output error is simply an error calculation that is done to determine how far off a neural network's output was from the ideal network. This value is rarely used for any purpose other than a stepping stone on the way to the calculation of root mean square (RMS) error. Once all training sets have been used the RMS error can be calculated. This error acts as the global error for the entire neural network.
You will recall form Chapter 3 that there is a method provided by the neural network class named "calcError". This method has two primary responsibly. First it calculates the output error for each member of the training set. This error is allowed to grow until all of the training sets have been presented. Then the RMS error is calculated. The calculation of the RMS error is covered in the next section.
The second task that is accomplished by the "calcError" method is the backpropagation deltas are calculated. This aspect of the "calcError" method will be covered in much greater detail in Chapter 5. We will now examine how the output error for a training set member is calculated. The source code for the "calcError" method is shown below.
public void calcError(double ideal[]) {
int i, j;
final int hiddenIndex = inputCount;
final int outputIndex = inputCount + hiddenCount;
// clear hidden layer errors
for (i = inputCount; i < neuronCount; i++) {
error[i] = 0;
}
// layer errors and deltas for output layer
for (i = outputIndex; i < neuronCount; i++) {
error[i] = ideal[i - outputIndex] - fire[i];
globalError += error[i] * error[i];
errorDelta[i] = error[i] * fire[i] * (1 - fire[i]);
}
// hidden layer errors
int winx = inputCount * hiddenCount;
for (i = outputIndex; i < neuronCount; i++) {
for (j = hiddenIndex; j < outputIndex; j++) {
accMatrixDelta[winx] += errorDelta[i] * fire[j];
error[j] += matrix[winx] * errorDelta[i];
winx++;
}
accThresholdDelta[i] += errorDelta[i];
}
// hidden layer deltas
for (i = hiddenIndex; i < outputIndex; i++) {
errorDelta[i] = error[i] * fire[i] * (1 - fire[i]);
}
// input layer errors
winx = 0; // offset into weight array
for (i = hiddenIndex; i < outputIndex; i++) {
for (j = 0; j < hiddenIndex; j++) {
accMatrixDelta[winx] += errorDelta[i] * fire[j];
error[j] += matrix[winx] * errorDelta[i];
winx++;
}
accThresholdDelta[i] += errorDelta[i];
}
}
As you can see from the above source code the error is calculated by the first block of code in the method. The error calculation begins with the following lines.
// layer errors and deltas for output layer
for (i = outputIndex; i < neuronCount; i++) {
First we are going to loop through every neuron in the output layer. Each element in the neural network's output is compared against the ideal output. The difference between these two values is calculated. This value is then squared and stored. This value is then added to the output error, stored in the global error property, as well.
error[i] = ideal[i - outputIndex] - fire[i];
globalError += error[i] * error[i];
errorDelta[i] = error[i] * fire[i] * (1 - fire[i]);
}
At the end of processing we now have added the error of this training set element to the global error accumulator. This process continues for each of the training set elements. Once each element in the set has been calculated you are ready to calculate the RMS error. Calculation of the RMS error will be discussed in the next section.
You will also notice that the above method calculates the "responsibility" of each of the neurons in the other layers for the error. This will be used when the train method is called to adjust the weights of the neural network. This method uses the backpropagation method, which will be discussed in greater detail in Chapter 5. For now we will continue by examining how the RMS is calculated.
Root Mean Square (RMS) Error
The RMS error is the error that was displayed by Chapter 3's example as the neural networked trained. It is the RMS error that allows the neural network to know if enough training has taken place. The RMS error can be calculated at any time after the "calcErrors" method has been called. This calculation is done by the "getError" method. The "getError" method is shown here.
public double getError(int len) {
double err = Math.sqrt(globalError / (len * outputCount));
globalError = 0; // clear the accumulator
return err;
}
As you can see you must pass in the length of the training set. This is necessary because the RMS is an average. To take the average error across all training set elements, you must know the size of the training set. The RMS error is then calculated by dividing the global error by the product of the training set length and the number of output neurons. The square root of this ratio produces the RMS. Finally, after the RMS error has been calculated the globalError is set back to zero. This is done so that it can begin accumulating for a new error.
Error Calculation and Unsupervised Training
We have discussed how errors are calculated for supervised training. Errors must also be calculated for unsupervised training as well. How this is done may not be initially obvious. How can an error be calculated when no correct output is provided? The exact procedure by which this is done will be covered in Chapter 6 when the Kohonen neural network is discussed. For now we will simply highlight the major details of the process as well as compare and contrast unsupervised training error calculation to supervised training error calculation.
Most unsupervised neural networks are designed to calssify input data. The input data should be classified into one of the output neurons. The degree to which each output neuron fires for the input data is studied to produce an error for unsupervised training. Ideally we would like just one single neuron to fire at a high level for each member of the training set. If this is not the case we adjust the weights to the neuron with the highest firing, that is the winning neuron, consolidates its win. This training method causes more and more neurons to fire for the different elements in the training set.
