Chapter 2: Building Encog Neural Networks
Chapter 2: Building Encog Neural Networks
- What are Layers and Synapses?
- Encog Layer Types
- Encog Synapse Types
- Neural Network Properties
- Neural Network Logic
- Building with Layers and Synapses
Encog neural networks are made up of layers, synapses, properties and a logic definition. In this chapter we will examine the various types of layers and synapses supported by Encog. You will see how the layer and synapse types can be combined to create a variety of neural network types.
What are Layers and Synapses?
A layer is a collection of similar neurons. All neurons in the layer must share exactly the same characteristic as other neurons in this layer. A layer accepts a parameter that specifies how many neurons that layer is to have. Layers hold an array of threshold values. There is one threshold value for each of the neurons in the layer. The threshold values, along with the weight matrix form the long-term memory of the neural network. Some layers also hold context values that make up the short-term memory of the neural network.
A synapse is used to connect one layer to another. The synapses contain the weight matrixes used by the neural network. The weight matrixes hold the connection values between each of the neurons in the two layers that are connected by this synapse.
Every Encog neural network contains a neural logic class. This class defines how a neural network processes its layers and synapses. A neural logic class must implement the INeuralLogic interface. Every Encog neural network must have an INeuralLogic based logic class. Without such a class the network would not be able to process incoming data. INeuralLogic classes allow Encog to be compatible with a wide array of neural network types.
Some INeuralLogic classes require specific layer types. For the INeuralLogic classes to find these layers, the layers must be tagged. Tagging allows a type to be assigned to any layer in the neural network. Not all layers need to be tagged.
Neural network properties are stored in a collection of name-value pairs. They are stored in a simple IDictionary structure. Some INeuralLogic classes require specific parameters to be set for them to operate. These parameters are stored in the neural network properties.
Neural networks are constructed of layers and synapses. There are several different types of layers and synapses, provided by Encog. This chapter will introduce all of the Encog layer types and synapse types. We will begin by examining the Encog layer types.
Understanding Encog Layers
There are a total of three different layer types used by Encog. In this section we will examine each of these layer types. All three of these layer types implement the ILayer interface. As additional layer types are added to Encog, they will support the ILayer interface as well. We will begin by examining the ILayer interface.
Using the Layer Interface
The ILayer interface defines many important methods that all layers must support. Additionally, most Encog layers implement a constructor that initializes that unique type of layer. Listing 2.1 shows the ILayer interface.
Listing 2.1: The Layer Interface
public interface ILayer : ICloneable, IEncogPersistedObject
{
void AddNext(ILayer next);
void AddNext(ILayer next, SynapseType type);
void AddSynapse(ISynapse synapse);
INeuralData Compute(INeuralData
pattern);
IActivationFunction ActivationFunction
{
get;
set;
}
int NeuronCount
{
get;
set;
}
IList<ISynapse> Next
{
get;
}
ICollection<ILayer> NextLayers
{
get;
}
double[] Threshold
{
get;
set;
}
int X
{
get;
set;
}
int Y
{
get;
set;
}
int ID
{
get;
set;
}
bool HasThreshold
{
get;
}
bool IsConnectedTo(ILayer layer);
void Process(INeuralData pattern);
INeuralData Recur();
}
As you can see there are a number of methods that must be implemented to create a layer. We will now review some of the more important methods.
The AddNext method is used to connect another layer to this one. The next layer is connected with an ISynapse. There are two overloads to the AddNext method. The first allows you to simply specify the next layer and a WeightedSynapse is automatically created to connect the new layer. The second allows you to specify the next layer and use the SynapseType enumeration to specify what type of synapse you would like to connect the two layers. Additionally, the AddSynapse method allows you to simply pass in an already created ISynapse.
The Next property can be called to get an IList of the ISynapse objects used to connect to the next layers. Additionally the NextLayers property can be used to determine which layers this ILayer is connected to. To see if this Layer is connected to another specific ILayer call the IsConnectedTo method.
The Threshold property allows access to the threshold values for this layer. The threshold values are numeric values that change as the neural network is trained, together with the weight matrix values; they form the long-term memory of the neural network. Not all layers have threshold values; the HasThreshold property can be used to determine if a layer has threshold values.
The Activation property allows access to the activation function. Activation functions are mathematical functions that scale the output from a neuron layer. Encog supports many different activation functions. Activation functions will be covered in much greater detail in the next chapter.
Finally, the Compute method is provided that applies the activation function and does any other internal processing necessary to Compute the output from this layer. You will not usually call Compute directly, rather you will call the Compute method on the INetwork that this layer is attached to, and it will call the appropriate Compute functions for its various layers.
Using the Basic Layer
The BasicLayer implements the ILayer interface. The BasicLayer class has two primary purposes. First, many types of neural networks can be built completely from BasicLayer objects, as it is a very useful layer in its own right. Second, the BasicLayer provides the basic functionality that some other layers require. As a result, some of the other layers in Encog subclass are base on the BasicLayer class.
The most basic form of the BasicLayer constructor accepts a single integer parameter that specifies how many neurons this layer will have. This constructor creates a layer that uses threshold values and the hyperbolic tangent function as the activation function. For example, the following code creates three layers, with varying numbers of neurons.
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(2));
network.AddLayer(new BasicLayer(6));
network.AddLayer(new BasicLayer(1));
network.Structure.
network.Reset();
If you would like more control over the layer, you can use a more advanced constructor. The following constructor allows you to specify the activation function, as well as if threshold values should be used.
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(new ActivationSigmoid(),true,2));
network.AddLayer(new BasicLayer(new ActivationSigmoid(),true,6));
network.AddLayer(new BasicLayer(new ActivationSigmoid(),true,1));
network.Structure.
network.Reset();
The above code creates the same sort of network as the previous code segment; however, a sigmoid activation function is used. The true parameter means that threshold values should be used. Some neural network architectures do not use threshold values, while others do. As you progress through this book you will see both networks that use, as well as those that do not use threshold values.
The BasicLayer class is used for many neural network types in this book.
Using the Context Layer
The ContextLayer class implements a contextual layer. This layer allows the neural network to have a short-term memory. The context layer always remembers the last input values that were fed to it. This causes the context layer to always output what it originally received on the previous run of the neural network. This allows the neural network to remember the last data that was fed to it on the previous run. The context layer is always one iteration behind.
Context layers are usually used with a recurrent neural network. Recurrent neural networks do not feed the layers just forward. Layers will be connected back into the flow of the neural network. Chapter 12, “Recurrent Neural Networks”, will discuss recurrent neural networks in greater detail. Two types of neural network that make use of the ContextLayer are the Elman and Jordan neural networks. Elman and Jordan neural networks will also be covered in Chapter 12. The following code segment shows how to create a ContextLayer.
ILayer context = new ContextLayer(2);
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(1));
network.AddLayer(hidden = new BasicLayer(2));
hidden.AddNext(context, SynapseType.OneToOne);
context.AddNext(hidden);
network.AddLayer(new BasicLayer(1));
network.Structure.
The above code shows a ContextLayer used with regular BasicLayer objects. The output from the hidden layer in the above neural network not only goes to the output layer. The output from the hidden layer also is fed into the ContextLayer. A OneToOneSynapse is used to feed the ContextLayer. We simply want the context layer to remember the output from the hidden layer; we do not want any processing. A WeightedSynapse is fed out of the ContextLayer because we do want additional processing. We want the neural network to learn from the output of the ContextLayer.
These features allow the ContextLayer to be very useful for recognizing sequences of input data. The patterns are no longer mutually exclusive when you use a ContextLayer. If “Pattern A” is presented to the neural network, followed by “Pattern B”, it is much different than “Pattern B” being presented first. Without a context layer, the order would not matter.
Using the Radial Basis Function Layer
The RadialBasisFunctionLayer object implements a radial basis function (RBF) layer. This layer type is based on one or more RBF functions. A radial basis function reaches a peak and decreases quickly on both sides of the graph. One of the most common radial basis functions is the Gaussian function. The Gaussian function is the default option for the RadialBasisFunctionLayer class. You can see the Gaussian function in Figure 2.1.
Figure 2.1: The Gaussian Function
The above figure shows a graph of the Gaussian function. Usually several Gaussian functions are combined to create a RadialBasisFunctionLayer. Figure 2.2 shows a RadialBasisFunctionLayer being edited in the Encog Workbench. Here you can see that this layer is made up of multiple Gaussian functions.
Figure 2.2: An RBF Layer in Encog Workbench
The following code segment shows the RadialBasisFunctionLayer as part of a RBF neural network.
RadialBasisFunctionLayer rbfLayer;
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(new ActivationLinear(),false,2));
network.AddLayer(rbfLayer = new RadialBasisFunctionLayer(4), SynapseType.Direct);
network.AddLayer(new BasicLayer(1));
network.Structure.
network.Reset();
rbfLayer.
As you can see from the above code, the RBF layer is used as a hidden layer between two BasicLayer objects. RBF layers will be discussed in greater detail in Chapter 14, “Common Neural Network Patterns”.
Understanding Encog Synapses
In the previous section you saw how neural networks are made up of layers. Synapses are used to connect these layers together. Encog supports a variety of different synapse types. Synapses are used to carry information between the levels of a neural network. The synapses differ primarily in how the neurons are connected and what processing should be done on the information as it flows between levels.
Some of the synapse types supported by Encog make use of weight matrixes. A weight matrix allows the connections between each of the source layer neurons to have its connection weighted to the target neuron layer. By adjusting each of these weights, the neural network can learn.
In the next section you will learn about the synapse types that Encog supports. Any synapse that Encog makes use of must support the ISynapse interface. This interface is discussed in the next section.
The Synapse Interface
The ISynapse interface defines all of the essential methods that a class must support to function as a synapse. The ISynapse interface is shown in Listing 2.2.
Listing 2.2: The Synapse Interface
public interface ISynapse : IEncogPersistedObject
{
INeuralData Compute(INeuralData input);
ILayer FromLayer
{
get;
set;
}
int FromNeuronCount
{
get;
}
Matrix.Matrix WeightMatrix
{
get;
set;
}
int MatrixSize
{
get;
}
ILayer ToLayer
{
get;
set;
}
int ToNeuronCount
{
get;
}
SynapseType SynapseType
{
get;
}
bool IsSelfConnected
{
get;
}
bool IsTeachable
{
get;
}
}
As you can see there are a number of methods that must be implemented to create a synapse. We will now review some of the more important methods.
The FromLayer and ToLayer properties can be used to find the source and target layers for the neural synapse. The IsSelfConnected can also be used to determine if the synapse creates a self-connected layer. Encog also supports self-connected layers. A layer is self-connected if it has a self-connected synapse. A self-connected synapse is a synapse where the “from layer” and “to layer” are the same layer.
The WeightMatrix property allows access to the weight matrix for the neural network. A neural network that has a weight matrix is “teachable”, and the IsTeachable method will return true. The MatrixSize property can also be called to determine the size of the weight matrix.
Finally, the Compute method is provided that applies any synapse specific transformation, such as weight matrixes. You will not usually call Compute directly, rather you will call the Compute method on the INetwork that this layer is attached to, and it will call the appropriate Compute functions for its various synapses.
Constructing Synapses
Often the synapses are simply created in the background and the programmer is not really aware of what type of synapse is even being created. The AddLayer method of the BasicNetwork class automatically creates a new WeightedSynapse every time a new layer is added to the neural network.
The AddLayer method of the BasicNetwork class hides quite a bit of complexity. However, it is useful to see what is actually going on, and how the synapses are created. The following lines of code will show how to create a neural network “from scratch”, where every object that is needed to create the neural network is created by hand. The first step is to create a BasicNetwork object to hold the layers.
network = new BasicNetwork();
Next, we create three layers. Hidden, input and output layers are all created.
ILayer inputLayer = new BasicLayer(new ActivationSigmoid(),
true,2);
ILayer hiddenLayer = new BasicLayer(
new ActivationSigmoid(),true,2);
ILayer outputLayer = new BasicLayer(
new ActivationSigmoid(),true,1);
Two synapses are needed to connect these three layers together. One synapse holds the input to the hidden layer. The second synapse holds the hidden to the output layer. The following lines of code create these synapses.
ISynapse synapseInputToHidden =
new WeightedSynapse(inputLayer,
ISynapse synapseHiddenToOutput
= new WeightedSynapse(hiddenLayer,
These synapses can then be added to the two layers they originate from.
inputLayer.Next.Add(
hiddenLayer.Next.Add(
The BasicNetwork object should be informed what the input and output layer. Finally, the network structure should be finalized and the weight matrix and threshold values reset.
network.TagLayer(BasicNetwork.
network.TagLayer(BasicNetwork.
network.Structure.
network.Reset();
This section will discuss the different types of synapses supported by Encog. We will begin with the weighted synapse.
Using the WeightedSynapse Class
The weighted synapse is perhaps the most commonly used synapse type in Encog. The WeightedSynapse class is used by many different neural network architectures. Any place that a learning synapse is needed, the WeightedSynapse class is a good candidate. The WeightedSynapse connects every neuron in the source layer with every neuron in the target layer. Figure 2.3 shows a diagram of the weighted synapse.
Figure 2.3: The Weighted Synapse
This is the default synapse type for Encog. To create a weighted synapse object usually you will simply add a layer to the network. The default synapse type is the weighted synapse. You can also construct a weighted synapse object with the following line of code.
ISynapse synapse = new WeightedSynapse(from,to);
Once the weighted synapse has been created, it can be added to the next collection on the source layer's target.
Using the Weightless Synapse
The weightless synapse works very similar to the weighted synapse. The primary difference is that there are no weights in the weightless synapse. It provides a connection between each of the neurons in the source layer to every other neuron in the target layer. Figure 2.4 shows the weightless synapse.
Figure 2.4: The Weightless Synapse
The weightless synapse is implemented inside of the WeightlessSynapse class. The following line of code will construct a weightless synapse.
ISynapse synapse = new WeightlessSynapse(from,to);
The weightless synapse is used when you would like to fully connect two layers, but want the information to pass through to the target layer untouched. The weightless synapse is unteachable.
Using the OneToOne Synapse
The one to one synapse works very similar to the weightless synapse. Like the weightless synapse, the one to one synapse does not include any weight values. The primary difference is that every neuron in the source layer is connected to the corresponding neuron in the target layer. Each neuron is connected to only one other neuron. Because of this, the one to one synapse requires that the source and target layers have the same number of neurons. Figure 2.5 shows the one to one synapse.
Figure 2.5: The One to One Synapse
The following code segment shows how to construct a neural network that makes use of a one to one layer. The one to one layer is used in conjunction with a context layer.
ILayer context = new ContextLayer(2);
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(1));
network.AddLayer(hidden = new BasicLayer(2));
hidden.AddNext(context, SynapseType.OneToOne);
context.AddNext(hidden);
network.AddLayer(new BasicLayer(1));
network.Structure.
The one to one synapse is generally used to directly feed the values from the output of a layer to a context layer. However, it can serve any purpose where you would like to send a copy of the output of one layer to another similarly sized layer.
Using the Direct Synapse
The direct synapse is useful when you want to send a complete copy of the output from the source to every neuron in the target. Most layers are not designed to accept an array from every source neuron, so the number of layers that the direct synapse can be used with is limited. Currently, the only Encog layer type that supports the DirectSynapse is the RadialBasisFunctionLayer class. Figure 2.6 shows how the direct synapse works.
Figure 2.6: The Direct Synapse
The following code segment shows how to use the DirectSynapse.
RadialBasisFunctionLayer rbfLayer;
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(
new ActivationLinear(), false, 2));
network.AddLayer(rbfLayer = new RadialBasisFunctionLayer(4),
SynapseType.Direct);
network.AddLayer(new BasicLayer(1));
network.Structure.
network.Reset();
rbfLayer.
As you can see, the DirectSynapse is being used to feed a RadialBasisFunctionLayer.
Understanding Neural Logic
Every Encog neural network must contain a neural logic class. The INeuralLogic classes define how a neural network will process its layers and synapses. All neural logic classes must implement the INeuralLogic interface. By default a BasicNetwork class will make use of the SimpleRecurrentLogic logic class. This class can be used for both feedforward and simple recurrent networks. Because these are some of the most common neural network types in use, the SimpleRecurrentLogic class was chosen as the default.
The next few sections summarize the network logic classes provided by Encog.
The ART1Logic Class
The ART1Logic class is used to implement an adaptive resonance theory neural network. Adaptive Resonance Theory (ART) is a form of neural network developed by Stephen Grossberg and Gail Carpenter. There are several versions of the ART neural network, which are numbered ART-1, ART-2 and ART-3. The ART neural network is trained using either a supervised or unsupervised learning algorithm, depending on the version of ART being used. ART neural networks are used for pattern recognition and prediction. Encog presently supports ART1.
To create an ART1 neural network with Encog you should make use of the ART1Logic class. An example of an ART1 neural network will be provided in Chapter 14, “Common Neural Network Patterns”.
The BAMLogic Class
The BAMLogic class is used to implement a Bidirectional Associative Memory (BAM) network. The BAM network is a type of neural network developed by Bart Kosko in 1988. The BAM is a recurrent neural network that works similarly to and allows patterns of different lengths to be mapped bidirectionally to other patterns. This allows it to act as almost a two-way hash map. During its training, the BAM network is fed pattern pairs. The two halves of each pattern do not have to be of the same length. However, all patterns must be of the same overall structure. The BAM network can be fed a distorted pattern on either side and will attempt to map to the correct value.
The BoltzmannLogic Class
The BoltzmannLogic class is used to implement a Boltzmann machine neural network. A Boltzmann machine is a type of neural network developed by Geoffrey Hinton and Terry Sejnowski. It appears identical to a Hopfield neural network except it contains a random nature to its output. A temperature value is present that influences the output from the neural network. As this temperature decreases so does the randomness. This is called simulated annealing. Boltzmann networks are usually trained in an unsupervised mode. However, supervised training can be used to refine what the Boltzmann machine recognizes.
To create a Boltzmann machine neural network with Encog you should make use of the BoltzmannLogic class. An example of a Boltzmann neural network will be provided in Chapter 12, “Recurrent Neural Networks”.
The FeedforwardLogic Class
To create a feedforward with Encog the FeedforwardLogic class should be used. It is also possible to use the SimpleRecurrentLogic class as in place of the FeedforwardLogic class; however, the network will run slower. If there are no recurrent loops, the more simple FeedforwardLogic class should be used.
The feedforward neural network, or perceptron, is a type of neural network first described by Warren McCulloch and Walter Pitts in the 1940s. The feedforward neural network, and its variants, is the most widely used form of neural network. The feedforward neural network is often trained with the backpropagation training technique, though there are other more advanced training techniques, such as resilient propagation. The feedforward neural network uses weighted connections from an input layer to zero or more hidden layers, and finally to an output layer. It is suitable for many types of problems. Feedforward neural networks are used frequently in this book.
The HopfieldLogic Class
To create a Hopfield neural network with Encog, you should use the HopfieldLogic class. Dr. John Hopfield developed the Hopfield neural network in 1979. The Hopfield network is a single layer recurrent neural network. The Hopfield network always maintains a "current state" which is the current output of the neural network. The Hopfield neural network also has an energy property, which is calculated exactly the same as the temperature property of the Boltzmann machine. The Hopfield network is trained for several patterns. The state of the Hopfield network will move towards the closest pattern, thus "recognizing" that pattern. As the Hopfield network moves towards one of these patterns, the energy lowers.
To create a Hopfield neural network with Encog you should make use of the HopfieldLogic class. An example of a Hopfield neural network will be provided in Chapter 12, “Recurrent Neural Networks”.
The SimpleRecurrentLogic Class
To create a neural network where some layers are connected to context layers that connect back to previous layers, you should use the SimpleRecurrentLogic class. The Elman and Jordan neural networks are examples of the sort of networks where the SimpleRecurrentLogic class can be used. The SimpleRecurrentLogic class can also be used to implement a simple feedforward neural network, however, the FeedforwardLogic class will execute faster.
To create either an Elman or Jordan type of neural network with Encog you should make use of the SimpleRecurrentLogic class. Several examples of recurrent neural networks will be provided in Chapter 12, “Recurrent Neural Networks”.
The SOMLogic Class
To create a Self Organizing Map with Encog the SOMLogic class should be used. The Self Organizing Map (SOM) is a neural network type introduced by Teuvo Kohonen. SOM's are used to classify data into groups.
To create a SOM neural network with Encog you should make use of the SOMLogic class. An example of a SOM neural network will be provided in Chapter 9, “Unsupervised Training Methods”.
Understanding Properties and Tags
The BasicNetwork class also provides properties and tags to address the unique needs of different neural network logic types. Properties provide a set of name-value pairs that the neural logic can access. This is how you set properties about how the neural network should function. Tags allow individual layers to be identified. Some of the neural network logic types will affect layers differently. The layer tags allow the neural network logic to know which layer is which.
The following code shows several properties being set for an ART1 network.
BasicNetwork network = new BasicNetwork();
network.SetProperty(ARTLogic.
The first parameter specifies the name of the property. The neural network logic classes will define constants for properties that they require. The name of the property is a string.
The following code shows two network layers being tagged.
network.TagLayer(BasicNetwork.
Here multiple tags are being applied to the layerF1 and layerF2 layers. One layer can have multiple tags; however, a single tag can only be applied to one layer.
The BasicNetwork class does not keep a list of layers. The only way that layers actually “join” the neural network is either by being tagged, or linked through a synapse connection to a layer that is already tagged.
Building with Layers and Synapses
You are now familiar with all of the layer and synapse types supported by Encog. You will now be given a brief introduction to building ANNs with these neural network types. You will see how to construct several neural network types. They will be used to solve problems related to the XOR operator. For now, the XOR operator is a good enough introduction to several neural network architectures. We will see more interesting examples, as the book progresses. We will begin with the feedforward neural network.
Creating Feedforward Neural Networks
The feedforward neural network is one of the oldest types of neural networks still in common use. The feedforward neural network is also known as the perceptron. The feedforward neural network works by having one or more hidden layers sandwiched between an input and output layer. Figure 2.7 shows an Encog Workbench diagram of a feedforward neural network.
Figure 2.7: The Feedforward Neural Network
Listing 2.3 shows a simple example of a feedforward neural network learning to recognize the XOR operator.
Listing 2.3: Simple XOR Feedforward Neural Network
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Encog.Neural.Networks;
using Encog.Neural.Networks.Layers;
using Encog.Neural.Activation;
using Encog.Neural.Data.Basic;
using Encog.Neural.NeuralData;
using Encog.Neural.Networks.
using Encog.Neural.Data;
using Encog.Neural.Networks.
using ConsoleExamples.Examples;
namespace Encog.Examples.XOR.Resilient
{
public class XORResilient : IExample
{
public static ExampleInfo Info
{
get
{
ExampleInfo info = new ExampleInfo(
typeof(XORResilient),
"xor-rprop",
"XOR Operator with Resilient Propagation",
"Use RPROP to learn the XOR operator.");
return info;
}
}
/// <summary>
/// Input for the XOR function.
/// </summary>
public static double[][] XOR_INPUT ={
new double[2] { 0.0, 0.0 },
new double[2] { 1.0, 0.0 },
new double[2] { 0.0, 1.0 },
new double[2] { 1.0, 1.0 } };
/// <summary>
/// Ideal output for the XOR function.
/// </summary>
public static double[][] XOR_IDEAL = {
new double[1] { 0.0 },
new double[1] { 1.0 },
new double[1] { 1.0 },
new double[1] { 0.0 } };
/// <summary>
/// Program entry point.
/// </summary>
/// <param name="args">Not used.</param>
public void Execute(IExampleInterface app)
{
BasicNetwork network = new BasicNetwork();
network.AddLayer(new BasicLayer(
new ActivationSigmoid(), true, 2));
network.AddLayer(new BasicLayer(
new ActivationSigmoid(), true, 6));
network.AddLayer(new BasicLayer(
new ActivationSigmoid(), true, 1));
network.Structure.
network.Reset();
INeuralDataSet trainingSet =
new BasicNeuralDataSet(XOR_INPUT, XOR_IDEAL);
// train the neural network
ITrain train = new ResilientPropagation(
network, trainingSet);
int epoch = 1;
do
{
train.Iteration();
Console.WriteLine("Epoch #" + epoch
+ " Error:" + train.Error);
epoch++;
} while ((epoch < 5000) && (train.Error > 0.001));
// test the neural network
Console.WriteLine("Neural Network Results:");
foreach (INeuralDataPair pair in trainingSet)
{
INeuralData output = network.Compute(pair.Input);
Console.WriteLine(pair.Input[
+ ", actual=" + output[0] + ",ideal=" + pair.Ideal[0]);
}
}
}
}
As you can see from the above listing, it is very easy to construct a three-layer, feedforward neural network. Essentially, three new BasicLayer objects are created and added to the neural network with calls to the AddLayer method. Because no synapse type is specified, the three layers are connected together using the WeightedSynapse.
You will notice that after the neural network is constructed, it is trained. There are quite a few ways to train a neural network in Encog. Training is the process where the weights and thresholds are adjusted to values that will produce the desired output from the neural network. This example uses resilient propagation (RPROP) training. RPROP is the best choice for most neural networks to be trained with Encog. For certain special cases, some of the other training types may be more efficient.
// train the neural network
ITrain train = new ResilientPropagation(
network, trainingSet);
With the trainer setup we must now cycle through a bunch of iterations, or epochs. Each of these training iterations should decrease the “error” of the neural network. The error is the difference between the current actual output of the neural network and the desired output.
int epoch = 1;
do
{
train.Iteration();
Console.WriteLine("Epoch #" + epoch
+ " Error:" + train.Error);
epoch++
Continue training the neural network so long as the error rate is greater than one percent.
} while ((epoch < 5000) &&
(train.Error > 0.001));
Now that the neural network has been trained, we should test it. To do this, the same data that the neural network was trained with is presented to the neural network. The following code does this.
Console.WriteLine("Neural Network Results:");
foreach (INeuralDataPair pair in trainingSet)
{
INeuralData output = network.Compute(pair.Input);
Console.WriteLine(pair.Input[
+ ", actual=" + output[0] + ",ideal=" + pair.Ideal[0]);
}
This will produce the following output:
Epoch #1 Error:0.9902997764512583
Epoch #2 Error:0.6762359214192293
Epoch #3 Error:0.49572129129302844
Epoch #4 Error:0.49279160045197135
Epoch #5 Error:0.5063357328001542
Epoch #6 Error:0.502484567412553
Epoch #7 Error:0.4919515177527043
Epoch #8 Error:0.49157058621332506
Epoch #9 Error:0.48883664423510526
Epoch #10 Error:0.48977067420698456
Epoch #11 Error:0.4895238942630234
Epoch #12 Error:0.4870271073515729
Epoch #13 Error:0.48534672846811844
Epoch #14 Error:0.4837776485977757
Epoch #15 Error:0.48184530627656685
Epoch #16 Error:0.47980242878514856
Epoch #17 Error:0.47746641141708474
Epoch #18 Error:0.4748474362926616
Epoch #19 Error:0.47162728117571795
Epoch #20 Error:0.46807640808835427
...
Epoch #495 Error:0.010583637636670955
Epoch #496 Error:0.010748859630158925
Epoch #497 Error:0.010342203029249158
Epoch #498 Error:0.00997945501479827
Neural Network Results: 0.0,0.0,
actual=0.005418223644461675,
As you can see the error rate starts off high and steadily decreases. Finally, the patterns are presented to the neural network. As you can see, the neural network can handle the XOR operator. It does not produce the exact output it was trained with, but it is very close. The values 0.0054 and 0.0076 are very close to zero, just as 0.987 and 0.986 are very close to one.
For this network, we are testing the neural network with exactly the same data that the neural network was trained with. Generally, this is a very bad practice. You want to test the neural network on data that it was not trained with. This lets you see how the neural network is performing with new data that it has never processed before. However, the XOR function only has four possible combinations, and they all represent unique patterns that the network must be trained for. Neural networks presented later in this book will not use all of their data for training. Rather, they will be tested on data it has never been presented with before.
Creating Self-Connected Neural Networks
We will now look at self-connected neural networks. The Hopfield neural network is a good example of a self-connected, neural network. The Hopfield neural network contains a single layer of neurons. This layer is connected to itself. Every neuron on the layer is connected to every other neuron on the same layer. However, no two neurons are connected to themselves. Figure 2.8 shows a Hopfield neural network diagramed in the Encog Workbench.
Figure 2.8: The Hopfield Neural Network
Listing 2.4 shows a simple example of a Hopfield neural network learning to recognize various patterns.
Listing 2.4: Hopfield Neural Network
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using ConsoleExamples.Examples;
using Encog.Neural.NeuralData.
using Encog.Neural.Networks;
using Encog.Neural.Networks.Logic;
using Encog.Neural.Networks.Pattern;
namespace Encog.Examples.Hopfield.
{
public class HopfieldAssociate: IExample
{
public static ExampleInfo Info
{
get
{
ExampleInfo info = new ExampleInfo(
typeof(HopfieldAssociate),
"hopfield-associate",
"Hopfield Associates Patterns",
"Simple Hopfield neural network that learns to associate patterns.");
return info;
}
}
public const int HEIGHT = 10;
public const
int WIDTH = 10;
private IExampleInterface
app;
public String[][] PATTERN = {
new String[WIDTH] {
"O O O O O ",
" O O O O O",
"O O O O O ",
" O O O O O",
"O O O O O ",
" O O O O O",
"O O O O O ",
" O O O O O",
"O O O O O ",
" O O O O O" },
new String[WIDTH] {
"OO OO OO",
"OO OO OO",
" OO OO ",
" OO OO ",
"OO OO OO",
"OO OO OO",
" OO OO ",
" OO OO ",
"OO OO OO",
"OO OO OO" },
new String[WIDTH] {
"OOOOO ",
"OOOOO ",
"OOOOO ",
"OOOOO ",
"OOOOO ",
" OOOOO",
" OOOOO",
" OOOOO",
" OOOOO",
" OOOOO" },
new String[WIDTH] {
"O O O O",
" O O O ",
" O O O ",
"O O O O",
" O O O ",
" O O O ",
"O O O O",
" O O O ",
" O O O ",
"O O O O" },
new String[WIDTH] {
"OOOOOOOOOO",
"O O",
"O OOOOOO O",
"O O O O",
"O O OO O O",
"O O OO O O",
"O O O O",
"O OOOOOO O",
"O O",
"OOOOOOOOOO" } };
public String[][] PATTERN2 = {
new String[WIDTH] {
" ",
" ",
" ",
" ",
" ",
" O O O O O",
"O O O O O ",
" O O O O O",
"O O O O O ",
" O O O O O" },
new String[WIDTH] {
"OOO O O",
" O OOO OO",
" O O OO O",
" OOO O ",
"OO O OOO",
" O OOO O",
"O OO O O",
" O OOO ",
"OO OOO O ",
" O O OOO" },
new String[WIDTH] {
"OOOOO ",
"O O OOO ",
"O O OOO ",
"O O OOO ",
"OOOOO ",
" OOOOO",
" OOO O O",
" OOO O O",
" OOO O O",
" OOOOO" },
new String[WIDTH] {
"O OOOO O",
"OO OOOO ",
"OOO OOOO ",
"OOOO OOOO",
" OOOO OOO",
" OOOO OO",
"O OOOO O",
"OO OOOO ",
"OOO OOOO ",
"OOOO OOOO" },
new String[WIDTH] {
"OOOOOOOOOO",
"O O",
"O O",
"O O",
"O OO O",
"O OO O",
"O O",
"O O",
"O O",
"OOOOOOOOOO" } };
public BiPolarNeuralData ConvertPattern(
String[][] data, int index)
{
int resultIndex = 0;
BiPolarNeuralData result =
new BiPolarNeuralData(WIDTH * HEIGHT);
for (int row = 0; row < HEIGHT; row++)
{
for (int col = 0; col < WIDTH; col++)
{
char ch = data[index][row][col];
result.SetBoolean(resultIndex+
}
}
return result;
}
public void Display(
BiPolarNeuralData pattern1, BiPolarNeuralData pattern2)
{
int index1 = 0;
int index2 = 0;
for (int row = 0; row < HEIGHT; row++)
{
StringBuilder line = new StringBuilder();
for (int col = 0; col < WIDTH; col++)
{
if (pattern1.GetBoolean(index1++)
line.Append('O');
else
line.Append(' ');
}
line.Append(" -> ");
for (int col = 0; col < WIDTH; col++)
{
if (pattern2.GetBoolean(index2++)
line.Append('O');
else
line.Append(' ');
}
Console.WriteLine(line.
}
}
public void Evaluate(BasicNetwork hopfield,
String[][] pattern)
{
HopfieldLogic hopfieldLogic =
(HopfieldLogic)hopfield.Logic;
for (int i = 0; i < pattern.Length; i++)
{
BiPolarNeuralData pattern1 = ConvertPattern(pattern, i);
hopfieldLogic.CurrentState = pattern1;
int cycles = hopfieldLogic.RunUntilStable(
BiPolarNeuralData pattern2 =
(BiPolarNeuralData)
Console.WriteLine("Cycles until stable(max 100): "
+ cycles + ", result=");
Display(pattern1, pattern2);
Console.WriteLine("-----------
}
}
public void Execute(IExampleInterface app)
{
this.app = app;
HopfieldPattern pattern = new HopfieldPattern();
pattern.InputNeurons = WIDTH * HEIGHT;
BasicNetwork hopfield = pattern.Generate();
HopfieldLogic hopfieldLogic =
(HopfieldLogic)hopfield.Logic;
for (int i = 0; i < PATTERN.Length; i++)
{
hopfieldLogic.AddPattern(
}
Evaluate(hopfield, PATTERN);
Evaluate(hopfield, PATTERN2);
}
}
}
The Hopfield example begins by creating a HopfieldPattern class. The pattern classes allow for common types of neural networks to be constructed automatically. You simply provide the parameters about the type of neural network you wish to create, and the pattern takes care of setting up layers, synapses, parameters and tags.
HopfieldPattern pattern = new HopfieldPattern();
This Hopfield neural network is going to recognize graphic patterns. These graphic patterns are mapped on to grids. The number of input neurons will be the total number of cells in the grid. This is the width times the height.
pattern.InputNeurons = WIDTH * HEIGHT;
The Hopfield pattern requires very little input, just the number of input neurons. Other patterns will require more parameters. Now that the HopfieldPattern has been provided with all that it needs, the Generate method can be called to create the neural network.
BasicNetwork hopfield = pattern.Generate();
The logic object is obtained for the Hopfield network.
HopfieldLogic hopfieldLogic =
(HopfieldLogic)hopfield.Logic;
The logic class is used to add the patterns that the neural network is to be trained on. This is similar to the training seen in the last section, except it happens much faster for the simple Hopfield neural network.
for(int i=0;i<PATTERN.length;i++){
hopfieldLogic.AddPattern(
}
Now that the network has been “trained” we will test it. Just like in the last section, we will evaluate the neural network with the same data with which it was trained.
Evaluate(hopfield,PATTERN);
However, in addition to the data that the network has already been presented with, we will also present new data. This new data are distorted images of the data that the network was trained on. The network should be able to still recognize the patterns, even though they were distorted.
Evaluate(hopfield,PATTERN2);
The following shows the output of the Hopfield neural network. As you can see the Hopfield neural network is first presented with the patterns that it was trained on. The Hopfield network simply echoes these patterns. Next, the Hopfield neural network is presented with distorted versions of the patterns with which it was trained. As you can see from the code snippet below, the Hopfield neural network still recognizes the values.
Cycles until stable(max 100): 1, result=
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
----------------------
Cycles until stable(max 100): 1, result=
OO OO OO -> OO OO OO
OO OO OO -> OO OO OO
OO OO -> OO OO
OO OO -> OO OO
OO OO OO -> OO OO OO
OO OO OO -> OO OO OO
OO OO -> OO OO
OO OO -> OO OO
OO OO OO -> OO OO OO
OO OO OO -> OO OO OO
----------------------
Cycles until stable(max 100): 1, result=
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
----------------------
Cycles until stable(max 100): 1, result=
O O O O -> O O O O
O O O -> O O O
O O O -> O O O
O O O O -> O O O O
O O O -> O O O
O O O -> O O O
O O O O -> O O O O
O O O -> O O O
O O O -> O O O
O O O O -> O O O O
----------------------
Cycles until stable(max 100): 1, result=
OOOOOOOOOO -> OOOOOOOOOO
O O -> O O
O OOOOOO O -> O OOOOOO O
O O O O -> O O O O
O O OO O O -> O O OO O O
O O OO O O -> O O OO O O
O O O O -> O O O O
O OOOOOO O -> O OOOOOO O
O O -> O O
OOOOOOOOOO -> OOOOOOOOOO
----------------------
Cycles until stable(max 100): 2, result=
-> O O O O O
-> O O O O O
-> O O O O O
-> O O O O O
-> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
O O O O O -> O O O O O
----------------------
Cycles until stable(max 100): 2, result=
OOO O O -> OO OO OO
O OOO OO -> OO OO OO
O O OO O -> OO OO
OOO O -> OO OO
OO O OOO -> OO OO OO
O OOO O -> OO OO OO
O OO O O -> OO OO
O OOO -> OO OO
OO OOO O -> OO OO OO
O O OOO -> OO OO OO
----------------------
Cycles until stable(max 100): 2, result=
OOOOO -> OOOOO
O O OOO -> OOOOO
O O OOO -> OOOOO
O O OOO -> OOOOO
OOOOO -> OOOOO
OOOOO -> OOOOO
OOO O O -> OOOOO
OOO O O -> OOOOO
OOO O O -> OOOOO
OOOOO -> OOOOO
----------------------
Cycles until stable(max 100): 2, result=
O OOOO O -> O O O O
OO OOOO -> O O O
OOO OOOO -> O O O
OOOO OOOO -> O O O O
OOOO OOO -> O O O
OOOO OO -> O O O
O OOOO O -> O O O O
OO OOOO -> O O O
OOO OOOO -> O O O
OOOO OOOO -> O O O O
----------------------
Cycles until stable(max 100): 2, result=
OOOOOOOOOO -> OOOOOOOOOO
O O -> O O
O O -> O OOOOOO O
O O -> O O O O
O OO O -> O O OO O O
O OO O -> O O OO O O
O O -> O O O O
O O -> O OOOOOO O
O O -> O O
OOOOOOOOOO -> OOOOOOOOOO
----------------------
As you can see, the neural network can recognize the distorted values as well as those values with which it was trained. This is a much more comprehensive test than was performed in the previous section. This is because the network is evaluated with data that it has never seen before.
When the Hopfield neural network recognizes a pattern, it returns the pattern that it was trained with. This is called autoassociation.
The program code for the Evaluate method will now be examined. This shows how to present a pattern to the neural network.
public void Evaluate(BasicNetwork hopfield,
String[][] pattern)
{
First the logic object is obtained.
HopfieldLogic hopfieldLogic =
(HopfieldLogic)hopfield.Logic;
Loop over all of the patterns and present each to the neural network.
for (int i = 0; i < pattern.Length; i++)
{
BiPolarNeuralData pattern1 = ConvertPattern(pattern, i);
The pattern is obtained from the array and converted to a form that can be presented to the neural network. The graphic patterns are binary, either the pixel is on or it is off. To convert the image all displayed pixels are converted to the numbers. We are using bipolar numbers, so a display pixel is 1, a hidden pixel is -1.
The Hopfield neural network has a current state. The neurons will be at either 1 or -1 level. The current state of the Hopfield network is set to the pattern that we want to recognize.
hopfieldLogic.CurrentState = pattern1;
The Hopfield network will be run until it stabilizes. A Hopfield network will adjust its pattern until it no longer changes. At this point it has stabilized. The Hopfield neural network will stabilize on one of the patterns that it was trained on. The following code will run the Hopfield network until it stabilizes, up to 100 iterations.
int cycles = hopfieldLogic.RunUntilStable(
BiPolarNeuralData pattern2 =
(BiPolarNeuralData)
Once the network's state has stabilized it is displayed.
Console.WriteLine("Cycles until stable(max 100): "
+ cycles + ", result=");
Display(pattern1, pattern2);
Console.WriteLine("-----------
These are
just a few of the neural network types that can be constructed with
Encog. As the book progresses, you will learn many more.
Chapter Summary
Encog neural networks are made up of layers, synapses, properties and a neural logic class. This chapter reviewed each of these. A layer is a collection of similar neurons. A synapse connects one layer to another. Properties define unique qualities that one neural network type might have. The neural logic class defines how the output of the neural network should be calculated.
Activation functions are very important to neural networks. Activation functions scale the output from one layer before it reaches the next layer. The next chapter will discuss how Encog makes use of activation functions.
Questions for Review
1. What is the purpose of an Encog layer.
2. What is autoassociation? Wheat Encog neural network type makes use of it.
3. Should the same data that was used to train a neural network be used to evaluate it? Why or why not?
4. What is the role of the neural logic class?
5. What are properties used for in an Encog neural network?
Terms
Activation Function
Adaptive Resonance Theory
Autoassociation
Basic Layer
Bidirectional Associative Memory
Boltzmann Machine
Direct Synapse
Elman Neural Network
Hopfield Neural Network
Hyperbolic Tangent Activation Function
Jordan Neural Network
Layer Tag
Network Pattern
Neural Logic
Neural Network Logic
Neural Network Properties
One-to-One Synapse
Pattern
Radial Basis Activation Function
Radial Basis Function
Radial Basis Function Layer
Self-Connected Layer
Sigmoid Activation Function
Threshold Value
Weight Matrix
Weighted Synapse
Weightless Synapse
RSS
YouTube
Twitter
Facebook
github
LinkedIn
