The Hopfield Neural Network

Get the entire book!

The Hopfield neural network is perhaps the simplest of neural networks. The Hopfield neural network is a fully connected single layer autoassociative network. This means it has one single layer, with each neuron connected to every other neuron. In this chapter we will examine a Hopfield neural network with just four neurons. This is a network that is small enough that it can be easily understood, yet can recognize a few patterns. A Hopfield network, with connections, is shown in figure 2.6.

We will build an example program that creates the Hopfield network shown in Figure 2.6. A Hopfield neural network has every Neuron connected to every other neuron. This means that in a four Neuron network there are a total of four squared or 16 connections. However, 16 connections assume that every neuron is connected to itself as well. This is not the case in a Hopfield neural network, so the actual number of connections is 12.

Figure 2.6: A Hopfield Neural Network with 12 Connections

As we write an example neural network program, we will store the connections in an array. Because each neuron can potentially be connected to every other neuron a two dimensional array will be used. Table 2.1 shows the layout of such an array.

Table 2.1: Connections on a Hopfield neural network

The connection weights put into this array, also called a weight matrix, allow the neural network to recall certain patterns when presented. For example, the values shown in Table 2.2 show the correct values to use to recall the patterns 0101 and 1010. The method to create the values contained in Table 2.2 will be covered shortly. First you will be shown how the values in Table 2.2 are used to recall 0101 and 1010.

Table 2.2: Weights used to recall 0101 and 1010

Recalling Patterns

You have been told several times that the connection weight matrix shown in table 2.2 will correctly recall 0101 and 1010. You will now be shown exactly how a neural network is used to recall patterns. First we will take the example of presenting 0101 to the Hopfield network. To do this we present each input neuron, which in this case are also the output neurons, with the pattern. Each neuron will activate based upon the input pattern. For example, when neuron 1 is presented with 0101 its activation will be the sum of all weights that have a 1 in input pattern. For example, we can see from Table 2.2 that Neuron 1 has the following weights with all of the other neurons:

We must now compare those weights with the input pattern of 0101:

We will sum only the values that contain a 1 in the input pattern. Therefore the activation of the first neuron is –1 + -1, or –2. The activation of each neuron is shown below.

N1 = -1 + -1 = -2
N2 = 0 + 1 = 1
N3= -1 + -1 = -2
N4 = 1 + 0 = 1

Therefore, the output neurons, which are also the input neurons, will report the above activations. The final output vector would then be –2, 1, -2, 1. These values are meaningless without a threshold method. We said earlier that a threshold method determines what range of values will cause the neuron, in this case the output neuron, to fire. The threshold usually used for a Hopfield network, is any value greater than zero. So the following neurons would fire.

N1 activation is –2, would not fire (0)
N2 activation is 1, would fire (1)
N3 activation is –2, would not fire(0)
N4 activation is 1m would fire (1)

As you can see, we assign a binary 1 to all neurons that fired, and a binary 0 to all neurons that did not fire. The final binary output from the Hopfield network would be 0101. This is the same as the input pattern. An autoassociative neural network, such as a Hopfield network, will echo a pattern back if the pattern is recognized. The pattern was successfully recognized. Now that you have seen how a connection weight matrix can cause a neural network to recall certain patterns, you will be shown how the connection weight matrix was derived.

Deriving the Weight Matrix

You are probably wondering how the weight matrix, shown by Table 2.2, was derived. This section will show you how to create a weight matrix that can recall any number of patterns. First you should start with a blank connection weight matrix, as follows.

We will first train this neural network to accept the value 0101. To do this we must first calculate a matrix just for 0101, which is called 0101’s contribution matrix. The contribution matrix will then be added to the actual connection weight matrix. As additional contribution matrixes are added to the connection weight matrix, the connection weight is said to learn each of the new patterns.

First we must calculate the contribution matrix of 0101. There are three steps involved in this process. First we must calculate the bipolar values of 0101. Bipolar simply means that you are representing a binary string with –1’s and 1’s rather than 0’s and 1’s. Next we transpose and multiply the bipolar equivalent of 0101 by itself. Finally, we set all the values from the north-west diagonal to zero, because neurons do not connect to themselves in a Hopfield network. Lets take each step one at a time and see how this is done, starting with the bipolar conversion.

Step 1: Convert 0101 to Bipolar

Bipolar is nothing more than a way to represent binary values as –1’s and 1’s rather than zero and 1’s. This is done because binary has one minor flaw. Which is that 0 is NOT the inverse of 1. Rather –1 is the mathematical inverse of 1.

To convert 0101 to bipolar we convert all of the zeros to –1’s. This results in:

0 = -1
1 = 1
0 = -1
1 = 1

The final result is the array –1, 1, -1, 1. This array will be used by step 2 to begin to build the contribution matrix for 0101.

Step 2: Multiply –1, 1, -1, 1 by its Inverse

For this step we will consider –1, 1, -1, 1 to be a matrix.

Taking the inverse of this matrix we have.

We must now multiply these two matrixes. Appendix B, “Mathematical Background”, tells you where to get additional information about matrix mathematics. It is a relatively easy procedure, where the rows and columns are multiplied against each other, to result in:

Condensed, this the above results in the following matrix.

Now that we have successfully multiplied the matrix by its inverse we are ready for
step 3.

Step 3: Set the Northwest Diagonal to Zero

Mathematically speaking we are now going to subtract the identity matrix from the matrix we derived in step two. The net result is that the northwest diagonal gets set to zero. The real reason we do this is Hopfield networks do not have their neurons connected to themselves. So positions [0][0], [1][1], [2][2] and [3][3] in our two dimensional array, or matrix, get set to zero. This results in the final contribution matrix for the bit pattern 0101.

This contribution matrix can now be added to whatever connection weight matrix you already had. If you only want this network to recognize 0101, then this contribution matrix becomes your connection weight matrix. If you also wanted to recognize 1001, then you would calculate both contribution matrixes and add each value in their contribution matrixes to result in a combined matrix, which would be the connection weight matrix.

If this process seems a bit confusing, you might try looking at the next section where we actually build a program that builds connection weight matrixes. There the process is explained in a more Java-centric way.

Before we end the discussion of determination of the weight matrix, one small side effect should be mentioned. We went through several steps to determine the correct weight matrix for 0101. Any time you create a Hopfield network that recognizes a binary pattern; the network also recognizes the inverse of that bit pattern. You can get the inverse of a bit pattern by flipping all 0’s to 1’s and 1’s to zeros. The inverse of 0101 is 1010. As a result, the connection weight matrix we just calculated would also recognize 1010.