We will begin by examining what exactly a neural network is. A simple feedforward neural network can be seen in Figure 1.1. This diagram was created with the Encog Workbench. It is not just a diagram; this is an actual functioning neural network from Encog as you would actually edit it.
Figure 1.1: Simple Feedforward Neural Network
Networks can also become more complex than the simple network above. Figure 1.1 shows a recurrent neural network.
Figure 1.2: Simple Recurrent Neural Network
Looking at the above two neural networks you will notice that they are composed of layers, represented by the boxes. These layers are connected by lines, which represent synapses. Synapses and layers are the primary building blocks for neural networks created by Encog. The next chapter focuses solely on layers and synapses.
Before we learn to build neural networks with layers and synapses, let’s first look at what exactly a neural network is. Look at Figures 1.1 and 1.2. They are quite a bit different, but they share one very important characteristic. They both contain a single input layer and a single output layer. What happens between these two layers is very different, between the two networks. In this chapter, we will focus on what comes into the input layer and goes out of the output layer. The rest of the book will focus on what happens between these two layers.
Every neural network seen in this book will have, at a minimum, an input and output layer. In some cases, the same layer will function as both input and output layer. You can think of the general format of any neural network found in this book as shown in Figure 1.3.
Figure 1.3: Generic Form of a Neural Network
To adapt a problem to a neural network, you must determine how to feed the problem into the input layer of a neural network, and receive the solution through the output layer of a neural network. We will look at the input and output layers in this chapter. We will then determine how to structure the input and interpret the output. The input layer is where we will start.
The input layer is the first layer in a neural network. This layer, like all layers, has a specific number of neurons in it. The neurons in a layer all contain similar properties. The number of neurons determines how the input to that layer is structured. For each input neuron, one double value is stored. For example, the following array could be used as input to a layer that contained five neurons.
double input = new double;
The input to a neural network is always an array of doubles. The size of this array directly corresponds to the number of neurons on this hidden layer. Encog uses the class NeuralData to hold these arrays. You could easily convert the above array into a NeuralData object with the following line of code.
NeuralData data = new BasicNeuralData(input);
The interface NeuralData defines any “array like” data that may be presented to Encog. You must always present the input to the neural network inside of a NeuralData object. The class BasicNeuralData implements the NeuralData interface. The class BasicNeuralData is not the only way to provide Encog with data. There are other implementations of NeuralData, as well. We will see other implementations later in the book.
The BasicNeuralData class simply provides a memory-based data holder for the neural network. Once the neural network processes the input, a NeuralData based class will be returned from the neural network's output layer. The output layer is discussed in the next section.
The output layer is the final layer in a neural network. The output layer provides the output after all of the previous layers have had a chance to process the input. The output from the output layer is very similar in format to the data that was provided to the input layer. The neural network outputs an array of doubles.
The neural network wraps the output in a class based on the NeuralData interface. Most of the built in neural network types will return a BasicNeuralData class as the output. However, future, and third party, neural network classes may return other classes based other implementations of the NeuralData interface.
Neural networks are designed to accept input, which is an array of doubles, and then produce output, which is also an array of doubles. Determining how to structure the input data, and attaching meaning to the output, are two of the main challenges to adapting a problem to a neural network. The real power of a neural network comes from its pattern recognition capabilities. The neural network should be able to produce the desired output even if the input has been slightly distorted.
As previously discussed, neural networks contain and input layer and an output layer. Sometimes the input layer and output layer are the same. Often the input and output layer are two separate layers. Additionally, other layers may exist between the input and output layers. These layers are called hidden layers. These hidden layers can be simply inserted between the input and output layers. The hidden layers can also take on more complex structures.
The only purpose of the hidden layers is to allow the neural network to better produce the expected output for the given input. Neural network programming involves first defining the input and output layer neuron counts. Once you have defined how to translate the programming problem into the input and output neuron counts, it is time to define the hidden layers.
The hidden layers are very much a “black box”. You define the problem in terms of the neuron counts for the hidden and output layers. How the neural network produces the correct output is performed, in part, by the hidden layers. Once you have defined the structure of the input and output layers you must define a hidden layer structure that optimally learns the problem. If the structure of the hidden layer is too simple it may not learn the problem. If the structure is too complex, it will learn the problem but will be very slow to train and execute.
Later chapters in this book will discuss many different hidden layer structures. You will learn how to pick a good structure, based on the problem that you are trying to solve. Encog also contains some functionality to automatically determine a potentially optimal hidden layer structure. Additionally, Encog also contains functions to prune back an overly complex structure. Chapter 13, “Pruning and Structuring Networks” shows how Encog can help create a potentially optimal structure.
Some neural networks have no hidden layers. The input layer may be directly connected to the output layer. Further, some neural networks have only a single layer. A single layer neural network has the single layer self-connected. These connections permit the network to learn. Contained in these connections, called synapses, are individual weight matrixes. These values are changed as the neural network learns. We will learn more about weight matrixes in the next chapter.