You are here

Error Calculation

    Error calculation is an important aspect of any neural network. Whether the neural network is supervised or unsupervised, an error rate must be calculated. The goal of virtually all training algorithms is to minimize the rate of error. In this section, we will examine how the rate of error is calculated for a supervised neural network. We will also discuss how the rate of error is determined for an unsupervised training algorithm. We will begin this section by examining two error calculation steps used for supervised training.

Error Calculation and Supervised Training

    There are two values that must be considered in determining the rate of error for supervised training. First, we must calculate the error for each element of the training set as it is processed. Second, we must calculate the average of the errors for all of the elements of the training set across each sample. For example, consider the XOR logical operator from chapter 1 that has only four items in its training set. Refer to Table 1.3 to review the XOR logical operator.

    In chapter 1, we intuitively chose values for the weight matrix. This is fine for a simple neural network, such as the one that implements the XOR operator. However, this is not practical for more complex neural networks. Typically, the process involves creating a random weight matrix and then testing each row in the training set. An output error is then calculated for each element of the training set. Finally, after all of the elements of the training set have been processed, the root mean square (RMS) error is determined for all of them.

Output Error

    The output error is simply an error calculation that is performed to determine how different a neural network’s output is from the ideal output. This value is rarely used for any purpose other than as a steppingstone in the calculation of the root mean square (RMS) error for the entire training set. Once all of the elements of a training set have been run through the network, the RMS error can be calculated. This error acts as the global rate of error for the entire neural network.

    We will create a generic error calculation class that will be used in all of the neural networks in this book. This class is named ErrorCalculation. This class works by calculating the output error for each member of the training set. This error is allowed to grow until all of the elements of the training set have been presented. Then, the RMS error is calculated. The calculation of the RMS error is covered in the next section. The ErrorCalculation class is shown in Listing 4.1.

Listing 4.1: The ErrorCalculation Class (ErrorCalculation.cs)

// Introduction to Neural Networks for C#, 2nd Edition
// Copyright 2008 by Heaton Research, Inc. 
// http://www.heatonresearch.com/online/introduction-neural-networks-cs-edition-2
//         
// ISBN13: 978-1-60439-009-4  	 
// ISBN:   1-60439-009-3
//   
// This class is released under the:
// GNU Lesser General Public License (LGPL)
// http://www.gnu.org/copyleft/lesser.html

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace HeatonResearchNeural.Util
{
    public class ErrorCalculation
    {
        /// <summary>
        /// The current error level.
        /// </summary>
        private double globalError;

        /// <summary>
        /// The size of a training set.
        /// </summary>
        private int setSize;

        /// <summary>
        /// Returns the root mean square error for a complete training set.
        /// </summary>
        /// <returns>The current error for the neural network.</returns>
        public double CalculateRMS()
        {
            double err = Math.Sqrt(this.globalError / (this.setSize));
            return err;

        }

        /// <summary>
        /// Reset the error accumulation to zero.
        /// </summary>
        public void Reset()
        {
            this.globalError = 0;
            this.setSize = 0;
        }

        /// <summary>
        /// Called to update for each number that should be checked.
        /// </summary>
        /// <param name="actual">The actual number.</param>
        /// <param name="ideal">The ideal number.</param>
        public void UpdateError(double[] actual, double[] ideal)
        {
            for (int i = 0; i < actual.Length; i++)
            {
                double delta = ideal[i] - actual[i];
                this.globalError += delta * delta;
                this.setSize += ideal.Length;
            }
        }
    }
}

    First, we will see how to use the ErrorCalculation class. When an error is to be calculated, there will usually be two arrays, the ideal array and the actual array. The ideal array contains the values that we hope the neural network will produce. An ideal array is shown here:

double[][] ideal = {
  new double[4] {1,2,3,4},
  new double[4] {5,6,7,8},
  new double[4] {9,10,11,12},
  new double[4] {13,14,15,16} };

    This ideal array contains four sets, which are the rows of the array. Each row contains four numbers. The neural network that this array would be used to train would, therefore, have four output neurons. The columns correspond to the output neurons. The rows are the individual elements of the training set.

    The actual output of the neural network is stored in an actual array. If we were training a neural network, we would have an array of input values that would result in the actual output values. A hypothetical actual array is provided here for use with the above ideal array.

double[][] actual = {
  new double[4] {1,2,3,5},
  new double[4] {5,6,7,8},
  new double[4] {9,10,11,12},
  new double[4] {13,14,15,16} };

    As you can see, the actual array is fairly close to the ideal array. An ErrorCalculation object is now instantiated named error.

ErrorCalculation error = new ErrorCalculation();

    The output for each element in the training set must now be compared to the ideal output. We loop through all four rows in the arrays.

for(int i=0;i<ideal.Length;i++) {
  error.UpdateError(actual[i], ideal[i]);
}

    The corresponding rows of the actual array and the ideal array are presented to the error object and the error object is updated.

    Finally, the RMS error is calculated and printed.

Console.WriteLine( error.CalculateRMS());

    The error object can be reused for another error calculation, if needed. Simply call the Reset method and the error object is ready to be used again.

    This CalculateError class will be used frequently in this book. Any time the RMS error is needed for a neural network, this class will be used. The next section will describe how the RMS error is calculated.

Root Mean Square (RMS) Error

    The RMS method is used to calculate the rate of error for a training set based on predefined ideal results. The RMS method is effective in calculating the rate of error regardless of whether the actual results are above or below the ideal results. To calculate the RMS for a series of n values of x, consider Equation 4.1.

Equation 4.1: Root Mean Square Error (RMS)

Supervised training.

    The values of x are squared and their sum is divided by n. Squaring the values eliminates the issue associated with some values being above the ideal values and others below, since computing the square of a value always results in a positive number.

    To apply RMS to the output of a neural network, consider Equation 4.2.

Equation 4.2: RMS for a Neural Network

Supervised training.

    To calculate the RMS for the arrays in the previous section, you would calculate the difference between the actual results and the ideal results, as shown in the above equation. The square of each of these would then be calculated and the results would be summed. The sum would then be divided by the number of elements, and the square root of the result of that computation would provide the rate of error.

    To implement this in C#, the UpdateError method is called to compare the output produced for each element of the training set with the ideal output values for the neural network. The signature for the UpdateError method is shown here:

public void UpdateError(double[] actual, double[] ideal)
{

    First, we loop through all of the elements in the actual array.

  for (int i = 0; i < actual.Length; i++)
  {

    We determine the difference, or delta, between the actual and the ideal values. It does not matter if this is a negative number; that will be handled in the next step.

  double delta = ideal[i] - actual[i];

    We then add the square of each delta to the globalError variable. The setSize variable is used to track how many elements have been processed.

  this.globalError += delta * delta;
  this.setSize += ideal.Length;
}

    Finally, once all of the elements in the training set have been cycled through the UpdateError method, the CalculateRMS method is called to calculate the RMS error. The signature for the CalculateRMS method is shown here.

public double CalculateRMS() 
{

    We calculate the error as the square root of the globalError divided by the setSize, and return the error.

  double err = Math.Sqrt(this.globalError / (this.setSize));
  return err;
}

    Once the CalculateError object has been used to calculate the rate of error, it must be reset before another training set can be processed. Otherwise, the globalError variable would continue to grow, rather than start from zero. To reset the CalculateError class, the Reset method should be called.

Error Calculation and Unsupervised Training

    We have discussed how errors are calculated for supervised training, we must now discuss how they are calculated for unsupervised training. This may not be immediately obvious. How can an error be calculated when no ideal outputs are provided? The exact procedure by which this is done will be covered in chapter 11, “Using a Self-Organizing Map.” For now, we will simply highlight the most important details of the process.

    Most unsupervised neural networks are designed to classify input data. The input data is classified based on one of the output neurons. The degree to which each output neuron fires for the input data is studied in order to produce an error for unsupervised training. Ideally, we would like a single neuron to fire at a high level for each member of the training set. If this is not the case, we adjust the weights to the neuron with the greatest number of firings, that is, the winning neuron consolidates its win. This training method causes more and more neurons to fire for the different elements in the training set.

Calais Document Category: 
Technology: 

Comments

gfoidl's picture

Shouldn't be the update of the setSize outside the loop?
I mean when update the value inside the loop we have n^2 elements instead of n element for computing the RMS. Or the update could be done inside the loop by adding 1 to the setSize.

BTW: Excellent book.

Kind regards,
Günther

philsenogun's picture

You are right Gunther. It confused me too

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer