Recalling Patterns

    You will now be shown exactly how a neural network is used to recall patterns. We will begin by presenting 0101 to the Hopfield network. To do this, we present each input neuron, which in this case are also the output neurons, with the pattern. Each neuron will activate based upon the input pattern. For example, when Neuron 1 is presented with 0101, its activation will result in the sum of all weights that have a 1 in the input pattern. For example, we can see from Table 3.2 that Neuron 1 has connections to the other neurons with the following weights:

    We must now compare those weights with the input pattern of 0101:

    We will sum only the weights corresponding to the positions that contain a 1 in the input pattern. Therefore, the activation of the first neuron is –1 + –1, or –2. The results of the activation of each neuron are shown below.

N1 = -1 + -1 = -2
N2 = 0 + 1 = 1
N3 = -1 + -1 = -2
N4 = 1 + 0 = 1

    Therefore, the output neurons, which are also the input neurons, will report the above activation results. The final output vector will then be –2, 1, –2, 1. These values are meaningless without an activation function. We said earlier that a threshold establishes when a neuron will fire. A threshold is a type of activation function. An activation function determines the range of values that will cause the neuron, in this case the output neuron, to fire. A threshold is a simple activation function that fires when the input is above a certain value.

    The activation function used for a Hopfield network is any value greater than zero, so the following neurons will fire. This establishes the threshold for the network.

N1 activation result is –2; will not fire (0)
N2 activation result is 1; will fire (1)
N3 activation result is –2; will not fire(0)
N4 activation result is 1; will fire (1)

    As you can see, we assign a binary value of 1 to all neurons that fired, and a binary value of 0 to all neurons that did not fire. The final binary output from the Hopfield network will be 0101. This is the same as the input pattern. An autoassociative neural network, such as a Hopfield network, will echo a pattern back if the pattern is recognized. The pattern was successfully recognized. Now that you have seen how a connection weight matrix can cause a neural network to recall certain patterns, you will be shown how the connection weight matrix was derived.

Deriving the Weight Matrix

    You are probably wondering how the weight matrix shown in Table 3.2 was derived. This section will explain how to create a weight matrix that can recall any number of patterns. First you should start with a blank connection weight matrix, as described in Equation 3.1.

Equation 3.1: A Blank Matrix

    We will first train this neural network to accept the value 0101. To do this, we must first calculate a matrix just for 0101, which is called 0101’s contribution matrix. The contribution matrix will then be added to the connection weight matrix. As additional contribution matrixes are added to the connection weight matrix, the connection weight is said to learn each of the new patterns.

    We begin by calculating the contribution matrix of 0101. There are three steps involved in this process. First, we must calculate the bipolar values of 0101. Bipolar representation simply means that we are representing a binary string with –1’s and 1’s, rather than 0’s and 1’s. Next, we transpose and multiply the bipolar equivalent of 0101 by itself. Finally, we set all the values from the northwest diagonal to zero, because neurons do not have connections to themselves in a Hopfield network. Let’s take the steps one at a time and see how this is done. We will start with the bipolar conversion.

    Step 1: Convert 0101 to its bipolar equivalent.

    We convert the input, because the binary representation has one minor flaw. Zero is NOT the inverse of 1. Rather –1 is the mathematical inverse of 1. Equation 3.2 can be used to convert the input string from binary to bipolar.

Equation 3.2: Binary to Bipolar

    Conversely, Equation 3.3 can be used to convert from bipolar to binary.

Equation 3.3: Bipolar to Binary

    To convert 0101 to its bipolar equivalent, we convert all of the zeros to –1’s, as follows:

    The final result is the array –1, 1, –1, 1. This array will be used in step 2 to begin building the contribution matrix for 0101.

    Step 2: Multiply –1, 1, –1, 1 by its transposition.

    For this step, we will consider –1, 1, –1, 1 to be a matrix:

Equation 3.4: Input Matrix

    Taking the transposition of this matrix we have:

Equation 3.5: Input Matrix Transposed

    We must now multiply these two matrixes. Matrix multiplication was covered in chapter 2. It is a relatively easy procedure in which the rows and columns are multiplied against each other, resulting in the following:

    Condensed, the above results in the following matrix:

Equation 3.6: Resulting Matrix

    Now that we have successfully multiplied the matrix by its inverse, we are ready for step 3.

    Step 3: Set the northwest diagonal to zero.

    Mathematically speaking, we are now going to subtract the identity matrix from the matrix we derived in step 2. The net result is that the cells in the northwest diagonal get set to zero. The real reason we do this is that neurons do not have connections to themselves in Hopfield networks. Thus, positions [0][0], [1][1], [2][2], and [3][3] in our two dimensional array, or matrix, get set to zero. This results in the final contribution matrix for the bit pattern 0101.

Equation 3.7: Contribution Matrix

    This contribution matrix can now be added to the connection weight matrix. If we only want this network to recognize 0101, then this contribution matrix becomes our connection weight matrix. If we also want to recognize 1001, then we would calculate both contribution matrixes and add the results to create the connection weight matrix.

    If this process seems a bit confusing, you might try looking at the next section in which we actually develop a program that builds connection weight matrixes. The process is explained using Java terminology.

    Before we end the discussion of determining the weight matrix, one small side effect should be mentioned. We went through several steps to determine the correct weight matrix for 0101. Any time you create a Hopfield network that recognizes a binary pattern, the network also recognizes the inverse of that bit pattern. You can get the inverse of a bit pattern by flipping all 0’s to 1’s and 1’s to 0’s. The inverse of 0101 is 1010. As a result, the connection weight matrix we just calculated would also recognize 1010.