The Weight Matrix

    In the last chapter, you learned that neural networks make use of two types of values: weights and thresholds. Weights define the interactions between the neurons. Thresholds define what it will take to get a neuron to fire. The weighted connections between neurons can be thought of as a matrix. For example, consider the connections between the following two layers of the neural network shown in Figure 2.1.

Figure 2.1: A two neuron layer connected to a three neuron layer.

    You can see the weights in Figure 2.1. The weights are attached to the lines drawn between the neurons. Each of the two neurons in the first layer is connected to each of the three neurons in the second layer. There are a total of six connections. These connections can be represented as a 3x2 weight matrix, as described in Equation 2.1.

Equation 2.1: A Weight Matrix

    The weight matrix can be defined in C# as follows:

Matrix weightMatrix = new Matrix(3,2);

    The threshold variable is not multidimensional, like the weight matrix. There is one threshold value per neuron. Each neuron in the second layer has an individual threshold value. These values can be stored in an array of C# double variables. The following code shows how the entire memory of the two layers can be defined.

Matrix weightMatrix = new Matrix(3,2);
double[] thresholds = new double[2];

    These declarations include both the 3x2 matrix and the two threshold values for the second layer. There is no need to store threshold values for the first layer, since it is not connected to another layer. Weight matrix and threshold values are only stored for the connections between two layers, not for each layer.

    The preferred method for storing these values is to combine the thresholds with the weights in a combined matrix. The above matrix has three rows and two columns. The thresholds can be thought of as the fourth row of the weight matrix, which can be defined as follows:

Matrix weightMatrix = new Matrix(4,2);

    The combined threshold and weight matrix is described in Equation 2.2. In this equation, the variable w represents the cells used to store weights and the variable t represents the cells used to hold thresholds.

Equation 2.2: A Threshold and Weight Matrix

    Combining the thresholds and weights in one matrix has several advantages. This matrix now represents the entire memory of this layer of the neural network and you only have to deal with a single structure. Further, since many of the same mathematical operations performed on the weight matrix are also performed on the threshold values, having them contained in a single matrix allows these operations to be performed more efficiently.