You are here

Neural Network Structure

Get the entire book!
Introduction to Neural Networks with Java

To study neural networks you must first become aware of their structure. A neural network is composed of several different elements. Neurons are the most basic unit. Neurons are interconnected. These connections are not equal, as each connection has a connection weight. Groups of networks come together to form layers. In this section we will explore each of these topics.

The Neuron

The neuron is the basic building block of the neural network. A neuron is a communication conduit that both accepts input and produces output. The neuron receives its input either from other neurons or the user program. Similarly, the neuron sends its output to other neurons or the user program.

When a neuron produces output, that neuron is said to activate, or fire. A neuron will activate when the sum of its inputs satisfies the neuron’s activation function. Consider a neuron that is connected to a number of other neurons. The variable “w” represents the weights between this neuron and the other “k” neurons. We will say that this neuron is connected to “k” other neurons. The variable “x” represents the input to this neuron from each of the other neurons. Therefore we must calculate the sum of every input x multiplied by the corresponding weight “w”. This is shown in the following equation. This book will use some mathematical notation to explain how the neural networks are constructed. Often this is theoretical and not absolutely necessary to use neural networks. A review of the mathematical concepts used in this book is covered in Appendix B, “Mathematical Background”.

Equation 2.1: Summing the Weight Matrix

Summing the Weight Matrix

This sum must be given to the neurons activation function. An activation function is just a simple Java method that tells the neuron if it should fire or not. For example, if you chose to have your neuron only activate when the input to that neuron is between 5 and 10, the following activation method might be used.

boolean thresholdFunction(double input)
  if( (input>=5) && (input<=10) )
    return true;
    return false;

The above method will return true if the neuron would have activated, false otherwise. The method simply checks to see if the input is between 5 and 10 and returns true upon success. Methods such as this are commonly called threshold methods (or sometimes threshold functions). The threshold for this neuron is any input between 5 and 10. A neuron will always activate when the input causes the threshold to be reached.

There are several threshold methods that are commonly used by neural networks. Chapter 3 will explore several of these threshold methods. The example given later in this chapter uses an activation method called the Hyperbolic Tangent, or TANH. It is not critical to understand exactly what a Hyperbolic Tangent is in order to use such a method. The TANH activation method is just one, of several, activation methods that you may use. Chapter 3 will introduce other activation methods and explain when each is used.

The TANH activation method will be fed the sum of the input patterns and connection weights, as previously discussed. This sum will be referred to as “u”. The TANH activation method simply returns the hyperbolic tangent of “u”. Unfortunately Java does not contain a hyperbolic tangent method. The formula to calculate the hyperbolic tangent of the variable “u” is shown below.

Equation 2.2: The TANH Function

A hyperbolic tangent activation can easily be written in Java, even with out a hyperbolic tangent method. The following Java code implements the above formula.

  public double tanh (double u)
    double a = Math.exp( u );
    double b = Math.exp( -u );
    return ((a-b)/(a+b));

The hyperbolic tangent threshold method will return values according to Figure 2.1. As you can see this gives it a range of numbers both greater than and less than zero. You will find that you will use the TANH threshold method when you must have output greater than and less than zero. If only positive numbers are needed, then the Sigmoid threshold method will be used. Choosing an activation method is covered in much greater detail in Chapter 3.

Figure 2.1: Hyperbolic Tangent (TANH)

Hyperbolic Tangent (TANH)

Neuron Connection Weights

The previous section already mentioned that neurons are usually connected together. These connections are not equal, and can be assigned individual weights. These weights are what give the neural network the ability to recognize certain patterns. Adjust the weights, and the neural network will recognize a different pattern.

Adjustment of these weights is a very important operation. Later chapters will show you how neural networks can be trained. The process of training is adjusting the individual weights between each of the individual neurons until we achieve close to the desired output.

Neuron Layers

Neurons are usually grouped into layers. Layers are groups of neurons that perform similar functions. There are three types of layers. The input layer is the layer of neurons that receive input from the user program. The layer of neurons that send data to the user program is the output layer. Between the input layer and output layer can be zero or more hidden layers. Hidden layer neurons are connected only to other neurons and never directly interact with the user program.

Figure 2.2 shows a neural network with one hidden layer. Here you can see the user program sends a pattern to the input layer. The input layer presents this pattern to the hidden layer. The hidden layer then presents information on to the output layer. Finally the user program collects the pattern generated by the output layer. You can also see the connections, which are formed between the neurons. Neuron 1 (N1) is connected to both neuron 5 (N5) and Neuron 6 (N6).

Figure 2.2: Neural Network Layers

Neural Network Layers

The input and output layers are not just there as interface points. Every neuron in a neural network has the opportunity to affect processing. Processing can occur at any layer in the neural network.

Not every neural network has this many layers. The hidden layer is optional. The input and output layers are required, but it is possible to have one layer act as both an input and output layer. Later, in this chapter you will be shown a Hopfield neural network. This is a single layer (combined input and output) neural network.

Now that you have seen how a neural network is constructed you will be shown how neural networks are used in pattern recognition. Finally, this chapter will conclude with an implementation of a single layer Hopfield neural network that can recognize a few basic patterns.


Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer