Now that you have been shown some of the basic concepts of neural network we will example an actual Java example of a neural network. The example program for this chapter implements a simple Hopfield neural network that you can used to experiment with Hopfield neural networks.

The example given in this chapter implements the entire neural network. More complex neural network examples will often use JOONE. JOONE will be introduced in Chapter 3. The complete source code to this, and all examples, can be found on the companion download from Heaton Research. To learn how to run the examples, refer to Appendix C, “Compiling Examples under Windows” and Appendix D, “Compiling Examples under Linux/UNIX”. These appendixes give thorough discussion of how to properly compile and execute examples. The classes used to create the Hopfield example are shown in Figure 2.7.

**Figure 2.7: Hopfield Example Classes**

You will now be shown a Java program that implements a 4-neuron Hopfield neural network. This simple program is implemented as a Swing Java Application. Figure 2.8 shows the application as it appears when it initially starts up. Initially, the network activation weights are all zero. The network has learned no patterns at this point.

**Figure 2.8: A Hopfield Example**

We will begin by teaching it to recognize the pattern 0101. Enter 0101 under the “Input pattern to run or train”. Click the “Train” button. Notice the weight matrix adjust to absorb the new knowledge. You should now see the same connection weight matrix as Figure 2.9.

**Figure 2.9: Training the Hopfield Network**

Now you will test it. Enter the pattern 0101 into the “Input pattern to run or train”(it should still be there from your training). The output will be “0101”. This is an autoassociative network, therefore it echoes the input if it recognizes it.

Now you should try something that does not match the training pattern exactly. Enter the pattern “0100” and click “Run”. The output will now be “0101”. The neural network did not recognize “0100”, but the closest thing it knew was “0101”. It figured you made an error typing and attempted a correction.

Now lets test the side effect mentioned previously. Enter “1010”, which is the binary inverse of what the network was trained with (“0101”). Hopfield networks always get trained for the binary inverse too. So if you enter “0101”, the network will recognize it.

We will try one final test. Enter “1111”, which is totally off base and not close to anything the neural network knows. The neural network responds with “0000”, it did not try to correct, it has no idea what you mean. You can play with the network more. It can be taught more than one pattern. As you train new patterns it builds upon the matrix already in memory. Pressing “Clear” clears out the memory.

Before we examine the portions of the Hopfield example application that are responsible for the actual neural network, we will first examine the user interface. The main application source code is shown in listing 2.1. This listing implements the Hopfield class, which is where the user interface code resides.

**Listing 2.1: The Hopfield Application (Hopfield.java)**

import java.awt.*; import javax.swing.*; import java.awt.event.*; /** * Example: The Hopfield Neural Network * * This is an example that implements a Hopfield neural * network. This example network contains four fully * connected neurons. This file, Hopfield, implements a * Swing interface into the other two neural network * classes: Layer and Neuron. * * @author Jeff Heaton * @version 1.0 */ public class Hopfield extends JFrame implements ActionListener { /** * The number of neurons in this neural network. */ public static final int NETWORK_SIZE = 4; /** * The weight matrix for the four fully connected * neurons. */ JTextField matrix[][] = new JTextField[NETWORK_SIZE][NETWORK_SIZE]; /** * The input pattern, used to either train * or run the neural network. When the network * is being trained, this is the training * data. When the neural network is to be ran * this is the input pattern. */ JComboBox input[] = new JComboBox[NETWORK_SIZE]; /** * The output from each of the four neurons. */ JTextField output[] = new JTextField[NETWORK_SIZE]; /** * The clear button. Used to clear the weight * matrix. */ JButton btnClear = new JButton("Clear"); /** * The train button. Used to train the * neural network. */ JButton btnTrain = new JButton("Train"); /** * The run button. Used to run the neural * network. */ JButton btnRun = new JButton("Run"); /** * Constructor, create all of the components and position * the JFrame to the center of the screen. */ public Hopfield() { setTitle("Hopfield Neural Network"); // create connections panel JPanel connections = new JPanel(); connections.setLayout( new GridLayout(NETWORK_SIZE,NETWORK_SIZE) ); for ( int row=0;row<NETWORK_SIZE;row++ ) { for ( int col=0;col<NETWORK_SIZE;col++ ) { matrix[row][col] = new JTextField(3); matrix[row][col].setText("0"); connections.add(matrix[row][col]); } } Container content = getContentPane(); GridBagLayout gridbag = new GridBagLayout(); GridBagConstraints c = new GridBagConstraints(); content.setLayout(gridbag); c.fill = GridBagConstraints.NONE; c.weightx = 1.0; // Weight matrix label c.gridwidth = GridBagConstraints.REMAINDER; //end row c.anchor = GridBagConstraints.NORTHWEST; content.add( new JLabel( "Weight Matrix for the Hopfield Neural Network:"),c); // Weight matrix c.anchor = GridBagConstraints.CENTER; c.gridwidth = GridBagConstraints.REMAINDER; //end row content.add(connections,c); c.gridwidth = 1; // Input pattern label c.anchor = GridBagConstraints.NORTHWEST; c.gridwidth = GridBagConstraints.REMAINDER; //end row content.add( new JLabel( "Click \"Train\" to train the following pattern:"),c); // Input pattern String options[] = { "0","1"}; JPanel inputPanel = new JPanel(); inputPanel.setLayout(new FlowLayout()); for ( int i=0;i<NETWORK_SIZE;i++ ) { input[i] = new JComboBox(options); inputPanel.add(input[i]); } c.gridwidth = GridBagConstraints.REMAINDER; //end row c.anchor = GridBagConstraints.CENTER; content.add(inputPanel,c); // Output pattern label c.anchor = GridBagConstraints.NORTHWEST; c.gridwidth = GridBagConstraints.REMAINDER; //end row content.add( new JLabel("Click \"Run\" to see output pattern:"),c); // Output pattern JPanel outputPanel = new JPanel(); outputPanel.setLayout(new FlowLayout()); for ( int i=0;i<NETWORK_SIZE;i++ ) { output[i] = new JTextField(3); output[i].setEditable(false); outputPanel.add(output[i]); } c.gridwidth = GridBagConstraints.REMAINDER; //end row c.anchor = GridBagConstraints.CENTER; content.add(outputPanel,c); // Buttons JPanel buttonPanel = new JPanel(); btnClear = new JButton("Clear"); btnTrain = new JButton("Train"); btnRun = new JButton("Run"); btnClear.addActionListener(this); btnTrain.addActionListener(this); btnRun.addActionListener(this); buttonPanel.setLayout(new FlowLayout()); buttonPanel.add(btnClear); buttonPanel.add(btnTrain); buttonPanel.add(btnRun); content.add(buttonPanel,c); // adjust size and position pack(); Toolkit toolkit = Toolkit.getDefaultToolkit(); Dimension d = toolkit.getScreenSize(); setLocation( (int)(d.width-this.getSize().getWidth())/2, (int)(d.height-this.getSize().getHeight())/2 ); setDefaultCloseOperation(WindowConstants.DISPOSE_ON_CLOSE); setResizable(false); } /** * Used to dispatch events from the buttons * to the handler methods. * * @param e The event */ public void actionPerformed(ActionEvent e) { if ( e.getSource()==btnRun ) run(); else if ( e.getSource()==btnClear ) clear(); else if ( e.getSource()==btnTrain ) train(); } /** * Called when the neural network is to be ran against * the input. */ protected void run() { boolean pattern[] = new boolean[NETWORK_SIZE]; int wt[][] = new int[NETWORK_SIZE][NETWORK_SIZE]; for ( int row=0;row<NETWORK_SIZE;row++ ) for ( int col=0;col<NETWORK_SIZE;col++ ) wt[row][col]=Integer.parseInt(matrix[row][col].getText()); for ( int row=0;row<NETWORK_SIZE;row++ ) { int i = input[row].getSelectedIndex(); if ( i==0 ) pattern[row] = false; else pattern[row] = true; } Layer net = new Layer(wt); net.activation(pattern); for ( int row=0;row<NETWORK_SIZE;row++ ) { if ( net.output[row] ) output[row].setText("1"); else output[row].setText("0"); if ( net.output[row]==pattern[row] ) output[row].setBackground(java.awt.Color.green); else output[row].setBackground(java.awt.Color.red); } } /** * Called to clear the weight matrix. */ protected void clear() { for ( int row=0;row<NETWORK_SIZE;row++ ) for ( int col=0;col<NETWORK_SIZE;col++ ) matrix[row][col].setText("0"); } /** * Called to train the weight matrix based on the * current input pattern. */ protected void train() { int work[][] = new int[NETWORK_SIZE][NETWORK_SIZE]; int bi[] = new int[NETWORK_SIZE]; for ( int x=0;x<NETWORK_SIZE;x++ ) { if ( input[x].getSelectedIndex()==0 ) bi[x] = -1; else bi[x] = 1; } for ( int row=0;row<NETWORK_SIZE;row++ ) for ( int col=0;col<NETWORK_SIZE;col++ ) { work[row][col] = bi[row]*bi[col]; } for ( int x=0;x<NETWORK_SIZE;x++ ) work[x][x] =-1; for ( int row=0;row<NETWORK_SIZE;row++ ) for ( int col=0;col<NETWORK_SIZE;col++ ) { int i = Integer.parseInt(matrix[row][col].getText()); matrix[row][col].setText( "" + (i+work[row][col])); } } /** * Main program entry point, display the * frame. * * @param args Command line arguments are not used */ static public void main(String args[]) { JFrame f = new Hopfield(); f.show(); } }

The most significant components are the matrix array, the input array and the output array. The weight matrix is stored in a two dimensional array of JTextField elements. It is here that you enter the weights for each neuron. Because there are four neurons in our sample applications, and each neuron is fully connected to the other four (including itself), a four by four two dimensional array is used.

In addition to the weight array, you must have an area where you can input new patterns for the application to either recognize or learn from. This is the purpose of the input array. The input array is a one-dimensional array that contains four elements. These four elements correspond to each of the four input neurons to the Hopfield neural network (the four input neurons are also the output neurons in this example). When you click the Train button, the weight matrix is adjusted so the pattern you stored in the input array will be recalled in the future. If you click the Run button, then the values stored in the input array will be presented to the Neural Network for possible recognition.

The output of this recognition is stored in the output array. If the pattern was successfully recognized, then the output array should exactly mimic the input array. If the input pattern was slightly different than the pattern the network was trained with, then the Neural Network attempts to find the closest match. Therefore, after an input pattern is presented to the Neural Network, the output array will reflect the Neural Network’s best guess at the pattern that was input to it.

How exactly the Neural Network derives this output will be discussed later. Next you will be shown how the matrix was determined. Determining the matrix that will produce the desired output is called training the network.

When the Train button is clicked, the application calls the method train. The train method is responsible for adjusting the weight matrix so that the new pattern can be correctly recalled. This process takes several steps. You will now be shown how the network trains.

The mathematical basis for training a Hopfield Neural Network was already explained earlier in the “Deriving the Weight Matrix” section. Now we will examine the process from a more programmatic basis. The first thing that the train method must do is retrieve the input pattern from the TextField controls. As the input is retrieved, the values are converted to bipolar values. As was discussed previously, conversion to bipolar simply involves converting every 0 to a –1 of a binary number; the output is stored in the “bi” array. This is done as follows.

// first convert to bipolar(0=-1, 1=1) for ( int x=0;x<4;x++ ) { if ( input[x].getSelectedIndex()==0 ) bi[x] = -1; else bi[x] = 1; }

The next step is to multiply the input sequence by its transposition. To see exactly what this means refer back to the “Deriving the Weight Matrix” section. To perform this operation, a new matrix is constructed, called work, that is perfectly square. Its width is determined by the width of the input sequence. In this case the work matrix is exactly 4X4, which is the same size as the weight matrix array shown on the application. Here, each element of the work matrix is filled with a value that is derived by multiplying the value of the input sequence that contains the same row and column. This is done with the following code.

// now multiply the matrix by its transposition for ( int row=0;row<4;row++ ) for ( int col=0;col<4;col++ ) { work[row][col] = bi[row]*bi[col]; }

A Hopfield Neural network does not generally assign weights between the same neurons. For example, there would be no weight between neuron 0 and itself. The weight of each of the four neurons back to itself must be set to –1 (0 in bipolar). This is done with the following code.

// next set the northwest diagonal to -1 for ( int x=0;x<4;x++ ) work[x][x] =-1;

Finally, the work matrix must be added to the existing weight matrix. This is done by taking each element of the work matrix and adding it to the corresponding element of the actual weight matrix. The result of this is displayed on the application.

// finally add to the existing weight matrix for ( int row=0;row<4;row++ ) for ( int col=0;col<4;col++ ) { int i = Integer.parseInt(matrix[row][col].getText()); matrix[row][col].setText( "" + (i+work[row][col])); }

You will now be shown how the application is able to recall a pattern. This is done by presenting the input sequence to the Neural Network and determining the output for each Neuron. The mathematical basis for this process was already described in the section “Recalling Patterns.” The application determines the output of the Neural Network by using the Layer class, shown in Listing 2.2.

**Listing 2.2: The Layer Class (Layer.java)**

public class Layer { /** * An array of neurons. */ protected Neuron neuron[] = new Neuron[4]; /** * The output of the neurons. */ protected boolean output[] = new boolean[4]; /** * The number of neurons in this layer. And because this is a * single layer neural network, this is also the number of * neurons in the network. */ protected int neurons; /** * A constant to multiply against the threshold function. * This is not used, and is set to 1. */ public static final double lambda = 1.0; /** * The constructor. The weight matrix for the * neurons must be passed in. Because this is * a single layer network the weight array should * always be perfectly square(i.e. 4x4). These * weights are used to initialize the neurons. * * @param weights A 2d array that contains the weights * between each * neuron and the other neurons. */ Layer(int weights[][]) { neurons = weights[0].length; neuron = new Neuron[neurons]; output = new boolean[neurons]; for ( int i=0;i<neurons;i++ ) neuron[i]=new Neuron(weights[i]); } /** * The threshold method is used to determine if the neural * network will fire for a given pattern. This threshold * uses the hyperbolic tangent (tanh). * * @param k The product of the neuron weights and the input * pattern. * @return Whether to fire or not to fire. */ public boolean threshold(int k) { double kk = k * lambda; double a = Math.exp( kk ); double b = Math.exp( -kk ); double tanh = (a-b)/(a+b); return(tanh>=0); } /** * This method is called to actually run the neural network. * * @param pattern The input pattern to present to the * neural network. */ void activation(boolean pattern[]) { int i,j; for ( i=0;i<4;i++ ) { neuron[i].activation = neuron[i].act(pattern); output[i] = threshold(neuron[i].activation); } } }

When the constructor is called for the Layer class, the weight matrix is passed in. This will allow the Layer class to determine what the output should be for a given input pattern. To determine the output sequence, an input sequence should be passed to the activation method of the Layer class. The activation method calls each of the neurons to determine their output. The following code does this.

for ( i=0;i<4;i++ ) { neuron[i].activation = neuron[i].act(pattern); output[i] = threshold(neuron[i].activation); }

The above loop stores each neuron’s activation value in the same neuron. Each activation is determined by calling the act method of the Neuron. To see how each neuron calculates its activation, we must first examine the Neuron class, which is shown in Listing 2.3.

**Listing 2.3: The Neuron Class (Neuron.java)**

public class Neuron { /** * The weights between this neuron and the other neurons on * the layer. */ public int weightv[]; /** * Activation results for this neuron. */ public int activation; /** * The constructor. The weights between this neuron and * every other neuron(including itself) is passed in as * an array. Usually the weight between this neuron and * itself is zero. * * @param in The weight vector. */ public Neuron(int in[]) { weightv = in; } /** * This method is called to determine if the neuron would * activate, or fire. * * @param x Neuron input * @return If the neuron would activate, or fire */ public int act(boolean x[] ) { int i; int a=0; for ( i=0;i<x.length;i++ ) if ( x[i] ) a+=weightv[i]; return a; } }

To calculate a neuron’s activation, that neuron simply sums all of its weight values. Only those weight values that have a 1 in the input pattern are calculated. This is accomplished by using the following code.

for ( i=0;i<x.length;i++ ) if ( x[i] ) a+=weightv[i]; return a;

Technology:

Events Facts:

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer