Implementing the Kohonen Neural Network
Get the entire book!
Now that you understand how the Kohonen neural network functions we will implement one using Java. In this section we will see how several classes can be used together to create a Kohonen neural network. Following this section you will be shown an example of how to use the Kohonen neural network classes to create a simple Kohonen neural network. Finally, in Chapter 7 you will be shown how to construct a more complex application based on the Kohonen neural network that can recognize handwriting.
First you must understand the structure of the Kohonen neural network classes that we are constructing. Many of the neural network examples in this book use the Java Object Oriented Neural Engine (JOONE). JOONE is constructed to primarily support supervised training. As a result, we will not use JOONE to construct the Kohonen neural network in this chapter.
- KohonenNetwork - Implements the methods that are unique to the Kohonen neural network. This is where the Kohonen neural network is trained and recalls patterns.
- Network - Contains the methods that are not unique to the Kohonen neural network. This class contains methods to calculate the dot product and vector length, amoung other common tasks.
- NeuralReportable - A simple interface that allows the Kohonen neural network to return progress information as the neural network is trained.
- TrainingSet - A training set holder object that can contain arrays of individual training items. The training set can hold both input and output elements. For the Kohonen neural network examples shown in this chapter only the input elements are used.
The classes that make up the Kohonen neural network can be expressed as a UML diagram. This allows you to see the relationships between the individual classes. This class structure is shown in Figure 6.3.
Figure 6.3: UML diagram of the Kohonen neural network classes
Now that you understand the overall structure of the Kohonen neural network classes we will examine each individual class. You will see how these classes work together to provide Kohonen network functionality. We will begin by examining how the training set is constructed using the TrainingSet class.
Training Sets
To train the Kohonen neural network training sets must be provided. This training data will be stored in the TrainingSet class. This class is designed to be a container class that could potiently be used for other neural network archetectures as well as the Kohonen neural network. The TrainingSet class is shown in Listing 6.1.
Listing 6.1: Storing Training Sets (TrainingSet.java)
import java.io.*;
import java.util.*;
public class TrainingSet {
protected int inputCount;
protected int outputCount;
protected double input[][];
protected double output[][];
protected int trainingSetCount;
TrainingSet ( int inputCount , int outputCount )
{
this.inputCount = inputCount;
this.outputCount = outputCount;
trainingSetCount = 0;
}
public int getInputCount()
{
return inputCount;
}
public int getOutputCount()
{
return outputCount;
}
public void setTrainingSetCount(int trainingSetCount)
{
this.trainingSetCount = trainingSetCount;
input = new double[trainingSetCount][inputCount];
output = new double[trainingSetCount][outputCount];
classify = new double[trainingSetCount];
}
public int getTrainingSetCount()
{
return trainingSetCount;
}
void setInput(int set,int index,double value)
throws RuntimeException
{
if ( (set<0) || (set>=trainingSetCount) )
throw(new RuntimeException("Training set out of range:" + set ));
if ( (index<0) || (index>=inputCount) )
throw(new RuntimeException("Training input index out of range:" + index ));
input[set][index] = value;
}
void setOutput(int set,int index,double value)
throws RuntimeException
{
if ( (set<0) || (set>=trainingSetCount) )
throw(new RuntimeException("Training set out of range:" + set ));
if ( (index<0) || (set>=outputCount) )
throw(new RuntimeException("Training input index out of range:" + index ));
output[set][index] = value;
}
double getInput(int set,int index)
throws RuntimeException
{
if ( (set<0) || (set>=trainingSetCount) )
throw(new RuntimeException("Training set out of range:" + set ));
if ( (index<0) || (index>=inputCount) )
throw(new RuntimeException("Training input index out of range:" + index ));
return input[set][index];
}
double getOutput(int set,int index)
throws RuntimeException
{
if ( (set<0) || (set>=trainingSetCount) )
throw(new RuntimeException("Training set out of range:" + set ));
if ( (index<0) || (set>=outputCount) )
throw(new RuntimeException("Training input index out of range:" + index ));
return output[set][index];
}
double []getOutputSet(int set)
throws RuntimeException
{
if ( (set<0) || (set>=trainingSetCount) )
throw(new RuntimeException("Training set out of range:" + set ));
return output[set];
}
double []getInputSet(int set)
throws RuntimeException
{
if ( (set<0) || (set>=trainingSetCount) )
throw(new RuntimeException("Training set out of range:" + set ));
return input[set];
}
}
As you can see from the above code the TrainingSet class is managing two variable length arrays. The first is the input vector. The input vector can be made up of one or more double numbers stored in the array. This input array can be accessed by two means.
First, the input training array can be accessed by using the method getInputSet and setInputSet methods. These methods allow access to the complete array of the input set. More commonly you will access the getInput and setInput methods. These methods can access an individual training element. To access an individual training set item you must provide two indexes. The first index specifies which training element should be accessed. In a large training set there may be many of these. The second index lets the TrainingSet object which indivual input element to access for the specified training item. In addition to the output elements the TrainingSet class also supports outputs. These outputs are accessed by calling the getOutput and setOutput methods.
The output methods are provided for completeness. These methods are not used by the Kohonen neural network. If the output array were used it would allow supervised training to take place as both the anticipated output and input would be provided.
Internally the TrainingSet class stores the following variables. Their use is summarized as follows.
- inputCount - The number of input elements there will be for each training sample.
- outputCount - The number of output elements there will be for each training sample.
- input[][] - The input training samples.
- output[][] - The output training samples.
- trainingSetCount - The number of training samples.
To use the training set with the Kohonen neural network the training set must be populated with data. Because the Kohonen neural network is unsupervised, only the input elements need be filled. Once a TrainingSet object has been constructed and properly populated with data, it will be passed to a KohonenNetwork object for training. This process will be explained later in this chapter with an example program.
Reporting Progress
For the Kohonen neural network to be effective it must be trained. This training process could potiently take awhile. Because of this the Kohonen classes were designed to support an interface, named NeuralReportable that allows updates to be transmitted between the Kohonen neural network and the external environment. This interface is shown in Listing 6.2.
Listing 6.2: Reporting Process (NeuralReportable.java)
public interface NeuralReportable
{
public void update(int retry,double totalError,double bestError);
}
An Java interface allows a class to state that it supports a particular set of methods. Any class that implements the NeuralReportable interface must include an update method. This allows the KohonenNetwork class to be able to call an update method from your class, even though the KohonenNetwork class has no other advance knowledge of your class.
The update method will be called for each training epoch. Three parameters are passed to the update method to indicate the progress of training. First a variable named retry is passed that tells you how many training cycles have passed. Secondly, two error results are returned. The first error, totalError, indicates the total error for the Kohonen neural network for the epoch that just occurred. The second error, bestError, indicates the best totalError that has occurred so far. Now that you have seen how to prepare training sets and receive status information from the training process you are ready to see how the Kohone network classes operate. We will begin by examining the network base class, which implements some basic methods that any neural network, the Kohonen network included, might have need of.
Network Base Class
We will now examine the Network class. This class is the base class for the KohonenNetwork class, which will ultimately provide the Kohonen neural network. The Network class provides some basic functionality that would be useful for networks other than just the Kohonen neural network. The network class is declared abstract so that it cannot be instantiated by itself. To make use of the Network class a child class must be derived. The KohonenNetwork class is provided as a child class. The KohonenNetwork class will be examined in the next section. The properties defined in the Network class are shown in Listing 6.3.
Listing 6.3: Network Base Class (Network.java)
import java.util.*;
abstract public class Network {
public final static double NEURON_ON=0.9;
public final static double NEURON_OFF=0.1;
protected double output[];
protected double totalError;
protected int inputNeuronCount;
protected int outputNeuronCount;
protected Random random = new Random(System.currentTimeMillis());
Each of these properties will be used to hold some attribute of the neural network, or a constant that will be used. The properties are used for the following purposes.
- NEURON_ON - The value that a neuron must be greater than, or equal to, to be considered "on".
- NEURON_OFF - The value that a neuron must be less than, or equal to, to be considered "off".
- output[] - The output from the neural network.
- totalError - The error from the last epoch. This is the total error, across all training sets.
- inputNeuronCount - The number of input neurons.
- outputNeuronCount - The number of output neurons.
- random - A random number generator, used to initialize the weights to random values.
Earlier we saw that calculating a vector length is an important part of the recollection process for a Kohonen neural network. The Network class contains a method that calculates the vector length for a given vector, expressed as an array. This method is shown in Listing 6.4.
Listing 6.4: Calculate a vector length (Network.java)
/**
* Calculate the length of a vector.
*
* @param v vector
* @return Vector length.
*/
static double vectorLength( double v[] )
{
double rtn = 0.0 ;
for ( int i=0;i<v.length;i++ )
rtn += v[i] * v[i];
return rtn;
}
As you can see, the vectorLength method sums the squares of the numbers contained in the vector. As discussed previously in this chapter, this is the method used to calculate a vector length. This method will be used by the Kohonen network class, shown in the next section.
Another important function provided by the Network base class is calculation of a dot product. The dot product was discussed previously in this chapter. The Kohonen network class, which will be discussed in the next section, makes use of the dot product method to calculate the output of the neural network. The dot product method is shown in Listing 6.5.
Listing 6.5: Calculate a dot product (Network.java)
/**
* Called to calculate a dot product.
*
* @param vec1 one vector
* @param vec2 another vector
* @return The dot product.
*/
double dotProduct(double vec1[] , double vec2[] )
{
int k,v;
double rtn;
rtn = 0.0;
k = vec1.length;
v = 0;
while ( (k--)>0 ) {
rtn += vec1[v] * vec2[v];
v++;
}
return rtn;
}
As you can see form the listing, the dot product method returns the sum of the products of each element in the vector.
The neuron weights are initially set to random values. These random values are then trained to produce better results. At the beginning of each training cycle the weights are also initialized to random values. The Network class provides a method to do this. Listing 6.6 shows a method that can be called to randomize the weights used by the neural network.
Listing 6.6: Randomize weights (Network.java)
/**
* Called to randomize weights.
*
* @param weight A weight matrix.
*/
void randomizeWeights( double weight[][] )
{
double r ;
int temp = (int)(3.464101615 / (2. * Math.random() ));
for ( int y=0;y<weight.length;y++ ) {
for ( int x=0;x<weight[0].length;x++ ) {
r = (double) random.nextInt(Integer.MAX_VALUE) + (double) random.nextInt(Integer.MAX_VALUE) -
(double) random.nextInt(Integer.MAX_VALUE) - (double) random.nextInt(Integer.MAX_VALUE) ;
weight[y][x] = temp * r ;
}
}
}
}
As you can see form the above listing the program simply loops through the entire weight matrix and assigns a random number. You will also notice that two random numbers are used for each calculation. This is intended to further randomize the weights beyond a single randomly generated number.
The KohonenNetwork Class
We will now examine the KohonenNetwork class. This class implements the Kohonen neural network. The KohonenNetwork class has several properties. These can be seen in Listing 6.7.
Listing 6.7: The Kohonen network properties (KohonenNetwork.java)
public class KohonenNetwork extends Network {
double outputWeights[][];
protected int learnMethod = 1;
protected double learnRate = 0.5;
protected double quitError = 0.1;
protected int retries = 10000;
protected double reduction = .99;
protected NeuralReportable owner;
public boolean halt = false;
protected TrainingSet train;
The properties used by the Kohonen network are summarized as follows.
- halt - Set this to true to abort training.
- learnMethod - The learning rate, set to 1 for subtractive or to another value for additive.
- learnRate - The initial learning rate.
- outputWeights[][] - The weights of the output neurons base on the input from the
- owner - The owner class, which must implement the NeuralReportable interface.
- quitError - Once the error rate reaches this level stop training.
- reduction - The amount to reduce the learnRate property by each epoch.
- retries - The total number of cycles to allow, this places a ceiling on the number of training cycles that can transpire.
- train - The training set.
The first of the Kohonen network methods that we will examine is the constructor. The constructor can be seen in Listing 6.8.
Listing 6.8: The Kohonen network constructor (KohonenNetwork.java)
/**
* The constructor.
*
* @param inputCount Number of input neurons
* @param outputCount Number of output neurons
* @param owner The owner object, for updates.
*/
public KohonenNetwork(int inputCount,int outputCount,NeuralReportable owner)
{
int n ;
totalError = 1.0 ;
this.inputNeuronCount = inputCount;
this.outputNeuronCount = outputCount;
this.outputWeights = new double[outputNeuronCount][inputNeuronCount+1];
this.output = new double[outputNeuronCount];
this.owner = owner;
}
As you can see the constructor sets up the wake matrix arrays. It also stores the number of input and output neurons in their properties. Management of these weight arrays is a very important aspect of implementing the Kohonen neural network. There are also several utility methods contained in the Network classes used specifically to manipulate these weight matrixes. These weight matrix manipulation utility methods can be seen in listing 6.9.
Listing 6.9: Weight utility methods
public static void copyWeights( KohonenNetwork dest , KohonenNetwork source )
{
for ( int i=0;i<source.outputWeights.length;i++ ) {
System.arraycopy(source.outputWeights[i],
0,
dest.outputWeights[i],
0,
source.outputWeights[i].length);
}
}
/**
* Clear the weights.
*/
public void clearWeights()
{
totalError = 1.0;
for ( int y=0;y<outputWeights.length;y++ )
for ( int x=0;x<outputWeights[0].length;x++ )
outputWeights[y][x]=0;
}
/**
* Called to initialize the Kononen network.
*/
public void initialize()
{
int i ;
double optr[];
clearWeights() ;
randomizeWeights( outputWeights ) ;
for ( i=0 ; i<outputNeuronCount ; i++ ) {
optr = outputWeights[i];
normalizeWeight( optr );
}
}
}
First you will notice that there is a method provided that will copy a weight matrix. It is important to be able to copy a weight matrix because as the Kohonen neural network is training itself it will need to copy weight matrices that supplant the current best case weight matrix. Many new weight matrix must replace old one to copy method is used.
You'll also notice that there is a method provided that is capable of completely blanking the weight matrix. This is done by setting every element in the weight matrix zero. Finally there is also an initialization method provided that will set all of the weight matrices to initial random values. Further, the initialization procedure will also normalize these weights as discussed earlier in the chapter. Weight normalization is performed by a special function named normalizeWeight that will be discussed later in this section.
There are several methods available that are used for normalization. Because normalization is an important part of the Kohonen neural network. Listing 6.10 shows a method is used to normalize the inputs into a Kohonen neural network.
Listing 6.10: Input normalization (KohonenNetwork.java)
/**
* Normalize the input.
*
* @param input input pattern
* @param normfac the result
* @param synth synthetic last input
*/
void normalizeInput(
final double input[] ,
double normfac[] ,
double synth[]
)
{
double length, d ;
length = vectorLength ( input ) ;
// just in case it gets too small
if ( length < 1.E-30 )
length = 1.E-30 ;
normfac[0] = 1.0 / Math.sqrt ( length ) ;
synth[0] = 0.0 ;
}
The exact process used to normalize an input vector was discussed earlier in this chapter. As you can see from the above code is method does not actually change the input values and anyway. Rather a normalization factor is returned after it is calculated. The normalization factor, as discussed earlier, is calculated by taking the reciprocal of the square root of the vector length.
The input vector is not the only thing the must be normalize for Kohonen neural network to function properly. Listing 6.11 shows how the weight matrix is also normalized.
Listing 6.11: Weight normalization (KohonenNetwork.java)
/**
* Normalize weights
*
* @param w Input weights
*/
void normalizeWeight( double w[] )
{
int i ;
double len ;
len = vectorLength ( w ) ;
// just incase it gets too small
if ( len < 1.E-30 )
len = 1.E-30 ;
len = 1.0 / Math.sqrt ( len ) ;
for ( i=0 ; i<inputNeuronCount ; i++ )
w[i] *= len ;
w[inputNeuronCount] = 0;
}
As you can see from the above code the weight normalization method begins by calculating the vector length of the weight fake vector that was passed into it. Using this vector length a normalization factor is then calculated. The weight normalization method then proceeds by multiplying this normalization factor against all of the weights were passed and in the weight vector. We've now see most of the basic utility methods that are provided by the coming neural network. We will not begin to examine some other methods that actually implement the training and recollection processes of the Kohonen neural network. We will begin by examining method that is used to present an input pattern to the Kohonen neural network. This method, named trial, a show in listing 6.12.
Listing 6.12: Input pattern trial (KohonenNetwork.java)
/**
* Try an input pattern. This can be used to present an input pattern
* to the network. Usually its best to call winner to get the winning
* neuron though.
*
* @param input Input pattern.
*/
void trial ( double input[] )
{
int i ;
double normfac[]=new double[1], synth[]=new double[1], optr[];
normalizeInput(input,normfac,synth) ;
for ( i=0 ; i<outputNeuronCount; i++ ) {
optr = outputWeights[i];
output[i] = dotProduct( input , optr ) * normfac[0]
+ synth[0] * optr[inputNeuronCount] ;
// Remap to bipolar (-1,1 to 0,1)
output[i] = 0.5 * (output[i] + 1.0) ;
// account for rounding
if ( output[i] > 1.0 )
output[i] = 1.0 ;
if ( output[i] < 0.0 )
output[i] = 0.0 ;
}
}
The exact process used to calculate the output from the Kohonen neural network was discussed earlier in this chapter. As you can see from the above code the method loops to room each of the output neurons and calculates the value for that output neurons. The value for each output neuron is calculated by taking the dot product of the normalized to input that are and the weights. Because of rounding errors the final output may be greater than 1 or less than 0. To account for these rounding errors less than 0 is treaded as 0 and greater than 1 is treated as 1. To final output for each neuron is stored in the output array.
As discussed earlier in this chapter we're not generally concerned with the output of each individual neuron. In the Kohonen neural network we are only interested in which neuron actually one. The winning neuron is the neuron that determines which class the network recalled the pattern has been part of. Listing 6.13 shows a method that is used to actually present an input pattern to the Kohonen network and receive the winning neuron. This method represents the main entry point into the Kohonen network for pattern classification.
Listing 6.13: Present a pattern and get the winning neuron (KohonenNetwork.java)
/**
* Present an input pattern and get the
* winning neuron.
*
* @param input input pattern
* @param normfac the result
* @param synth synthetic last input
* @return The winning neuron number.
*/
public int winner(double input[] ,double normfac[] ,double synth[])
{
int i, win=0;
double biggest, optr[];
normalizeInput( input , normfac , synth ) ; // Normalize input
biggest = -1.E30;
for ( i=0 ; i<outputNeuronCount; i++ ) {
optr = outputWeights[i];
output[i] = dotProduct (input , optr ) * normfac[0]
+ synth[0] * optr[inputNeuronCount] ;
// Remap to bipolar(-1,1 to 0,1)
output[i] = 0.5 * (output[i] + 1.0) ;
if ( output[i] > biggest ) {
biggest = output[i] ;
win = i ;
}
// account for rounding
if ( output[i] > 1.0 )
output[i] = 1.0 ;
if ( output[i] < 0.0 )
output[i] = 0.0 ;
}
return win ;
}
The process executed by the above code was described earlier in this chapter. Much of the code shown here is the same as in the trial method that we just examined. This method will generally be used when you actually want to present a pattern to the neural network for classification. The trial method that we just examined is generally used only internally by the Kohonen neural network during training. During training we do care about the actual output of each neuron, whereas during classification we only care which neuron actually won.
The winner method works by looping through each of the output neurons and calculating the output of that particular neuron. As the method loops through each of these output neurons and indexes saved to which ever neuron had the highest output. This neuron with the highest output is considered to be the winner. At this point the winning neuron is returned.
Now that we have examined how the classification phase is implemented we can begin to examine the training phase. There are several methods that are used to facilitate the training process. The main entry point that is called the training the neural network is the training method. The training method is shown in listing 6.14.
Listing 6.14: Train the neural network (KohonenNetwork.java)
/**
* This method is called to train the network. It can run
* for a very long time and will report progress back to the
* owner object.
*
* @exception java.lang.RuntimeException
*/
public void learn ()
throws RuntimeException
{
int i, key, tset,iter,n_retry,nwts;
int won[],winners ;
double work[],correc[][],rate,best_err,dptr[];
double bigerr[] = new double[1] ;
double bigcorr[] = new double[1];
KohonenNetwork bestnet; // Preserve best here
totalError = 1.0 ;
bestnet = new KohonenNetwork(inputNeuronCount,outputNeuronCount,owner) ;
won = new int[outputNeuronCount];
correc = new double[outputNeuronCount][inputNeuronCount+1];
if ( learnMethod==0 )
work = new double[inputNeuronCount+1];
else
work = null ;
rate = learnRate;
initialize () ;
best_err = 1.e30 ;
// main loop:
n_retry = 0 ;
for ( iter=0 ; ; iter++ ) {
evaluateErrors ( rate , learnMethod , won ,
bigerr , correc , work ) ;
totalError = bigerr[0] ;
if ( totalError < best_err ) {
best_err = totalError ;
copyWeights ( bestnet , this ) ;
}
winners = 0 ;
for ( i=0;i<won.length;i++ )
if ( won[i]!=0 )
winners++;
if ( bigerr[0] < quitError )
break ;
if ( (winners < outputNeuronCount) &&
(winners < train.getTrainingSetCount()) ) {
forceWin ( won ) ;
continue ;
}
adjustWeights ( rate , learnMethod , won , bigcorr, correc ) ;
owner.update(n_retry,totalError,best_err);
if ( halt ) {
owner.update(n_retry,totalError,best_err);
break;
}
Thread.yield();
if ( bigcorr[0] < 1E-5 ) {
if ( ++n_retry > retries )
break ;
initialize () ;
iter = -1 ;
rate = learnRate ;
continue ;
}
if ( rate > 0.01 )
rate *= reduction ;
}
// done
copyWeights( this , bestnet ) ;
for ( i=0 ; i<outputNeuronCount ; i++ )
normalizeWeight ( outputWeights[i] ) ;
halt = true;
n_retry++;
owner.update(n_retry,totalError,best_err);
}
The training method begins by initializing the weight matrices to random values and setting up other important variables. Once the initialization is complete the main loop can begin. The main loop presents the training patterns to the neural network and calculates errors based on the results obtained from the neural network. The winning neuron has been determined. Once the winning neuron has been determined to that neuron is trained to further consolidates its ability to recognize his particular pattern. At the end of the main loop it is determined it further training will results in a better weight matrix. This is determined by calculating the improvement in error between this epoch and the previous epoch. If this improvement is as to negligible sand this cycle is considered complete. To begin a new cycle we initialize the weight matrix to random values and continue with new epochs.
As training progresses we will likely encounter a situation where certain neurons never win, regardless of which pattern is being presented. We would like all output neurons to participate in the classification of the patter, so we also execute a method that assists non-winning neurons in better adapting to some of the training data. This process will be discussed later in this section.
As ever faster cycles we keep track of which cycle had best error rate. Once we find a weight matrix that is below the specified desired error level training is complete. If we never find a weight matrix is below the specified error level than training will be complete win the number of cycles succeeded the maximum number of cycles specified. If this happens then we will take the best weight matrix that was determined in previous cycles.
The processes of evaluating errors, adjusting awaits, forcing non-winning neurons to win all individual processes that are isolated and external methods. We will not examine each of these processes. We will begin by examining how the errors are evaluated. Listing 6.15 shows how to errors are evaluated.
Listing 6.15: Evaluate errors (KohonenNetwork.java)
/**
* This method does much of the work of the learning process.
* This method evaluates the weights against the training
* set.
*
* @param rate learning rate
* @param learn_method method(0=additive, 1=subtractive)
* @param won a Holds how many times a given neuron won
* @param bigerr a returns the error
* @param correc a returns the correction
* @param work a work area
* @exception java.lang.RuntimeException
*/
void evaluateErrors (
double rate ,
int learn_method ,
int won[],
double bigerr[] ,
double correc[][] ,
double work[])
throws RuntimeException
{
int best, size,tset ;
double dptr[], normfac[] = new double[1];
double synth[]=new double[1], cptr[], wptr[], length, diff ;
// reset correction and winner counts
for ( int y=0;y<correc.length;y++ ) {
for ( int x=0;x<correc[0].length;x++ ) {
correc[y][x]=0;
}
}
for ( int i=0;i<won.length;i++ )
won[i]=0;
bigerr[0] = 0.0 ;
// loop through all training sets to determine correction
for ( tset=0 ; tset<train.getTrainingSetCount(); tset++ ) {
dptr = train.getInputSet(tset);
best = winner ( dptr , normfac , synth ) ;
won[best]++;
wptr = outputWeights[best];
cptr = correc[best];
length = 0.0 ;
for ( int i=0 ; i<inputNeuronCo the unt ; i++ ) {
diff = dptr[i] * normfac[0] - wptr[i] ;
length += diff * diff ;
if ( learn_method!=0 )
cptr[i] += diff ;
else
work[i] = rate * dptr[i] * normfac[0] + wptr[i] ;
}
diff = synth[0] - wptr[inputNeuronCount] ;
length += diff * diff ;
if ( learn_method!=0 )
cptr[inputNeuronCount] += diff ;
else
work[inputNeuronCount] = rate * synth[0] + wptr[inputNeuronCount] ;
if ( length > bigerr[0] )
bigerr[0] = length ;
if ( learn_method==0 ) {
normalizeWeight( work ) ;
for ( int i=0 ; i<=inputNeuronCount ; i++ )
cptr[i] += work[i] - wptr[i] ;
}
}
bigerr[0] = Math.sqrt ( bigerr[0] ) ;
}
The evaluate errors method is used to evaluate how well the network is training and to create a correction array that contains the corrections that will be made by the adjustWeights method. The method works by presenting each of the training elements to the network. For each of these elements presented we track which output neuron "won". This winning neuron is then adjusted so that it will respond even better to the training set the next time. This causes neurons to consolidate their win.
An array, named won, is kept that keeps a count of how many times each neuron wins. Later this array will be used to help neurons that failed to ever win. This neurons will be reassigned to remove some of the burden from output neurons that are already winning "too much". This process will be discussed later in this chapter.
Once the correction has been calculated the weights must be adjusted by this correction. Listing 6.16 shows how the weights are adjusted.
Listing 6.16: Adjust weights (KohonenNetwork.java)
/**
* This method is called at the end of a training iteration.
* This method adjusts the weights based on the previous trial.
*
* @param rate learning rate
* @param learn_method method(0=additive, 1=subtractive)
* @param won a holds number of times each neuron won
* @param bigcorr holds the error
* @param correc holds the correction
*/
void adjustWeights (
double rate ,
int learn_method ,
int won[] ,
double bigcorr[],
double correc[][]
)
{
double corr, cptr[], wptr[], length, f ;
bigcorr[0] = 0.0 ;
for ( int i=0 ; i<outputNeuronCount ; i++ ) {
if ( won[i]==0 )
continue ;
wptr = outputWeights[i];
cptr = correc[i];
f = 1.0 / (double) won[i] ;
if ( learn_method!=0 )
f *= rate ;
length = 0.0 ;
for ( int j=0 ; j<=inputNeuronCount ; j++ ) {
corr = f * cptr[j] ;
wptr[j] += corr ;
length += corr * corr ;
}
if ( length > bigcorr[0] )
bigcorr[0] = length ;
}
// scale the correction
bigcorr[0] = Math.sqrt ( bigcorr[0] ) / rate ;
}
As you can see from the above code the weight array is adjusted a factor storage in the correction array. The correction is scaled by the learning rate.
Training occurs by adjusting the winning neuron. Sometimes there will be neurons that fail to ever win. Listing 6.17 shows the process that is used to force a neuron to win.
Listing 6.17: Force a winning neuron
/**
* If no neuron wins, then force a winner.
*
* @param won how many t but for him to a imes each neuron won
* @exception java.lang.RuntimeException
*/
void forceWin(
int won[]
)
throws RuntimeException
{
int i, tset, best, size, which=0;
double dptr[], normfac[]=new double[1];
double synth[] = new double[1], dist, optr[];
size = inputNeuronCount + 1 ;
dist = 1.E30 ;
for ( tset=0 ; tset<train.getTrainingSetCount() ; tset++ ) {
dptr = train.getInputSet(tset);
best = winner ( dptr , normfac , synth ) ;
if ( output[best] < dist ) {
dist = output[best] ;
which = tset ;
}
}
dptr = train.getInputSet(which);
best = winner ( dptr , normfac , synth ) ;
dist = -1.e30 ;
i = outputNeuronCount;
while ( (i--)>0 ) {
if ( won[i]!=0 )
continue ;
if ( output[i] > dist ) {
dist = output[i] ;
which = i ;
}
}
optr = outputWeights[which];
System.arraycopy(dptr,
0,
optr,
0,
dptr.length);
optr[inputNeuronCount] = synth[0] / normfac[0] ;
normalizeWeight ( optr ) ;
}
There are three steps that must be carried out to handle output neurons that are failing to ever learn. First we must loop through the entire training set and find the training set pattern the causes the least activation. This training set will be assigned to the variable which. This training set is considered to be the least well represented by the current winning neurons.
The second step is to choose an output neuron that will be modified to better classify the training set identified in the previous step. This is done by looping through every output neuron that did not ever win and seeing which one has the highest activation for the training pattern identified in step one. Finally, in the third step, we will modify the weight of this neuron so that it will better classify this pattern next time.
You have now seen how the Kohonen neural network was implemented. In the next section you will be shown a simple example program that tests the classes that were just presented. In Chapter 7 you will be shown a complete example that will recognize hand-written characters.
RSS
YouTube
Twitter
Facebook
github
LinkedIn
