Chapter 4: Using the Encog Workbench

Submitted by jeffheaton on Tue, 04/13/2010 - 18:42

Creating a Neural Network
Creating a Training Set
Training a Neural Network
Querying the Neural Network
Generating Code

An important part of the Encog Framework is the Encog Workbench. The Encog Workbench is a GUI application that can be used to create and edit neural networks. Encog can persist neural networks to .EG files. These files are an XML representation of the neural networks, and other information in which Encog uses to store data. You will learn more about how to use C# to load and save from .EG files in Chapter 7, “Encog Persistence”.

The Encog workbench can be downloaded from the following URL:

http://www.encog.org

There are several different ways that the Encog Workbench is packaged. Depending on your computer system, you should choose one of the following:

Universal – Packaged with shell scripts and batch files to launch the workbench under UNIX, Macintosh or Windows.
Windows Application – Packaged with a Windows launcher. Simply double click the application executable and the application will start.
Macintosh Application – Packaged with a Macintosh launcher. Simply double click the application icon and the application will start.

In this chapter I will assume that you are using the Windows Application package of Encog Workbench. The others will all operate very similarly. Once you download the Encog workbench and unzip it to a directory, the directory will look similar to Figure 4.1.

The Encog Workbench was implemented as a Java application. However, it is compatible with the .Net and Silverlight versions of Encog as well. Java was chosen as the language to write the Workbench in due to its ability to run on many different hardware platforms.

Figure 4.1: The Encog Workbench Folder

To launch the Encog workbench double click the “Encog Workbench” icon. This will launch the Encog Workbench application. Once the workbench starts, you will see something similar to what is illustrated in Figure 4.2.

Figure 4.2: The Encog Workbench Application

The Encog Workbench can run a benchmark to determine how fast Encog will run on this machine. This may take several minutes, as it runs Encog through a number of different neural network operations. The benchmark is also a good way to make sure that Encog is functioning properly on a computer. To run the benchmark, click the “Tools” menu and select “Benchmark Encog”. The benchmark will run and display a progress bar. Once the benchmark is done, you will see the final benchmark number. This can be seen in Figure 4.3.

Figure 4.3: Benchmarking Encog

A lower number reflects a better score. The number is the amount of seconds that it took Encog to complete the benchmark tasks. Each part of the benchmark is run multiple times to try to produce consistent benchmark numbers. Encog's use of multicore processors will be reflected in this number. If the computer is already running other processes, this will slow down the benchmark. Because of this, you should not have other applications running while performing a benchmark using the Encog Workbench.

Creating a Neural Network

We will begin by creating a neural network. The Encog Workbench starts with an empty file. Once objects have been added to this empty file, it can be saved to an .EG file. This .EG file can then be loaded by the workbench again or loaded by Java or .Net Encog applications. The .Net and Java versions of Encog read exactly the same type of .EG files.

To create a neural network, select “Create Object on the “Objects menu”. A small popup window will appear that asks for the type of object to create. Choose “Neural Network” to create a new neural network. This will bring up a window that lets you browse the available types of neural networks to create. These are predefined templates for many of the common neural network types supported by Encog. This window can be seen in Figure 4.4.

Figure 4.4: Create a Neural Network

You will notice that the first option is to create an “Empty Neural Network”. Any of the neural networks shown here could be created this way. You would simply create an empty network and add the appropriate layers, synapses, tags and properties to create the neural network type you wish to create. However, if you would like to create one of the common neural network types, it is much faster to simply use one of these predefined templates. Choose the “Feedforward Neural Network”. You will need to fill in some information about the type of feedforward neural network you would like to create. This dialog box is seen in Figure 4.5.

Figure 4.5: Create a Feedforward Neural Network

We are going to create a simple, neural network that learns the XOR operator. Such a neural network should be created as follows:

Input Neuron Count: 2
Hidden Layer 1 Neuron Count: 2
Output Neuron Count: 1

The two input neurons are necessary because the XOR operator takes two input parameters. The one output neuron is needed because the XOR operator takes one output parameter. This can be seen from the following truth table for the XOR operator.

0 XOR 0 = 0

1 XOR 0 = 1

0 XOR 1 = 1

1 XOR 1 = 0

As you can see from the code above, the XOR operator takes two parameters and produces one value. The XOR operator only returns true, or one, when the two input operators are different. This defines the input and output neuron counts.

The hidden layer count is two. The hidden neurons are necessary to assist the neural network in learning the XOR operator. Two is the minimum number of hidden neurons that can be provided for the XOR operator. You may be wondering how we knew to use two. Usually this is something of a trial and error process. You want to choose the minimum number of hidden neurons that still sufficiently solves the problem.

Encog can help with this trial and error process. This process is called pruning. You will learn about pruning, and other automated techniques for determine good hidden layer counts in Chapter 13, “Pruning and Structuring Networks”.

Now that the feedforward neural network has been created, you will see it in the workbench. Figure 4.6 shows the workbench with a neural network added.

Figure 4.6: Neural Network Added

If you double click the feedforward neural network shown in Figure 4.6, it will open. This allows you to see the layers and synapses. Figure 4.7 shows the feedforward neural network that was just created.

Figure 4.7: The Newly Created Neural Network

The above figure shows how neural networks are edited with Encog. You can add additional layers and synapses. You can also edit other aspects of the neural network, such as properties and the type of neural logic that it uses.

Now that the neural network has been created, a training set should be created. The training set will be used to train the neural network.

Creating a Training Set

A training set is a collection of data to be used to train the neural network. There are two types of training sets commonly used with Encog.

Supervised Training
Unsupervised Training

Supervised training data has both an input and expected output specified for the neural network. For example, a truth table above could be represented as a training set. There would be four rows, one for each of the combinations fed to the XOR operator. You would have two input columns and one output column. These correspond to the input and output neurons. The training sets are not concerned with hidden layers. Hidden layers are simply present to assist in learning.

Unsupervised training data only has input values. There are no expected outputs. The neural network will train, in an unsupervised way, and determine for itself what the outputs should be. Unsupervised training is often used for classification problems where you want the neural network to group input data.

First, we must create a training set. Select “Create Object” from the “Objects” menu. Select a training set. Once the training set has been created it will be added along with the network that was previously created.

Figure 4.8: The Newly Created Training Set

Double clicking the training set will open it. The training set will open in a spreadsheet style window, as seen in Figure 4.9.

Figure 4.9: Editing the Training Set

Here you can see the training set. By default, Encog creates a training set for XOR. This is just the default. Usually you would now create the desired number of input and output columns. However, because we are training the XOR operator, the data is fine as it is.

Training a Neural Network

Training a neural network is a process where the neural network's weights and thresholds are modified so that the neural network will produce output according to the training data. There are many different ways to train a neural network. The choice of training method will be partially determined by the neural network type you are creating. Not all neural network types work with all training methods.

To train the neural network open it as you did for Figure 4.7. Click the “Train” button at the top of the window. This will display a dialog box that allows you to choose a training method, as seen in Figure 4.10.

Figure 4.10: Choosing a Training Method

Choose the resilient training method, under propagation. This is usually the best training method available for a supervised feedforward neural network. There are several parameters you can set for the resilient training method. For resilient training it is very unlikely that you should ever change any of these options, other than perhaps the desired maximum error, which defaults to 1%. You can see this dialog box in Figure 4.11.

Figure 4.11: Resilient Propagation Training

Selecting OK will open a window that will allow you to monitor the training progress, as seen in Figure 4.12.

Figure 4.12: About to Begin Training

To begin training, click the “Start” button on the training dialog box. The network will begin training. For complex networks, this process can go on for days. This is a very simple network that will finish in several hundred iterations. You will not likely even see the graph begin as the training will complete in a matter of seconds. Once the training is complete, you will see the following screen.

Figure 4.13: Training Complete

The training is complete because the current error fell below the maximum error allowed that was entered in Figure 4.11, which is 1%. Now that the network has been trained it can produce meaningful output when queried. The training finished very quickly. As a result, there were not enough iterations to draw a chart to show the training progress.

Querying the Neural Network

Querying the neural network allows you to specify values for the inputs to the neural network and observe the outputs. To query the neural network, click “Query” at the top of the network editor seen in Figure 4.7. This will open the query window as seen in Figure 4.14.

Figure 4.14: Query the Neural Network

As you can see from the above window, you are allowed to enter two values for the input neurons. When you click “Calculate”, the output values will be shown. In the example above two zeros were entered, which resulted in 0.008. This is consistent with the XOR operator, as 0.008 is close to zero. To get a value even closer to zero, train the neural network to a lower error rate.

You can also view the weights and threshold values that were generated by the training. From the network editor, shown in Figure 4.7, right click the synapse and choose “Edit Weight Matrix” from the popup menu. Likewise, you can view the thresholds by right-clicking and choosing “Edit Layer” from the pop-up menu. Figure 4.15 shows the dialog used to edit the layer properties.

Figure 4.15: View Layer Properties

You can also browse available activation functions. If you choose to change the activation function you will see something similar to that shown in Figure 4.16.

Figure 4.16: Edit the Activation Function

In Figure 4.16 you can see that the current activation function is the hyperbolic tangent. The graph for the hyperbolic tangent function is also shown for reference.

Generating Code

The Encog workbench provides two ways that you can make use of your neural network in C# code. First, you can save the neural network and training data to an .EG file. C# applications can then load data from this .EG file. Using .EG files will be covered in much greater detail in Chapter 7, “Encog Persistence”.

Another way to generate code is to use the Encog Workbench. The Encog workbench can generate code in the following languages.

Java
C#
VB.Net

Code generation simply generates the code needed to create the neural network only. No code is generated to train or use the neural network. For the generated program to be of any use, you will need to add your own training code. Listing 4.1 shows the generated C# code from the XOR, feedforward neural network.

Listing 4.1: Generated C# Code

using Encog.Neural.Activation;

using Encog.Neural.Networks;

using Encog.Neural.Networks.Layers;

using Encog.Neural.Networks.Synapse;

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using org.encog.neural.networks.synapse.WeightedSynapse;

// Neural Network file generated by Encog. This file shows just // a simple neural network generated for the structure

// designed in the workbench.

// Additional code will be needed for training and processing.

// http://www.encog.org

namespace EncogSandbox

{

public class Program {

public static void Main()

{

BasicNetwork network = new BasicNetwork();

ILayer outputLayer = new BasicLayer(

new ActivationSigmoid(),true,1);

ILayer hiddenLayer1 = new BasicLayer(

new ActivationSigmoid(),true,2);

ILayer inputLayer = new BasicLayer(

new ActivationSigmoid(),true,2);

ISynapse synapse1 =

new WeightedSynapse(hiddenLayer1,outputLayer)

ISynapse synapse2 =

new WeightedSynapse(inputLayer,hiddenLayer1)

hiddenLayer1.AddSynapse(synapse1);

inputLayer.AddSynapse(synapse2);

network.TagLayer("INPUT",inputLayer);

network.TagLayer("OUTPUT",outputLayer);

network.Structure.FinalizeStructure();

network.Reset();

}

The same network could also have been generated in Java or VB.Net.

Summary

In this chapter you saw how to use the Encog Workbench. The Encog Workbench provides a way to edit the .EG files produced by the Encog Framework. There are also templates available to help you quickly create common neural network patterns. There is also a GUI network editor that allows networks to be designed using drag and drop functionality.

The workbench allows training data to be created as well. Training data can be manually entered or imported from a CSV file. Training data includes the input to the neural network, as well as the expected output. Training data that only includes input data will be used in unsupervised training. Training data that includes both input and expected output will be used in supervised training.

The neural network can be trained using many different training algorithms. For a feedforward neural network, one of the best choices is the resilient propagation algorithm. The Encog Workbench allows you to enter parameters for the training, and then watch the progress of the training.

The Encog Workbench will generate the code necessary to produce a neural network that was designed with it. The workbench can generate code in Java, C# or VB.Net. This code shows how to construct the neural network with the necessary layers, synapses, properties and layer tags.

The code generated in this chapter was capable of creating the neural network that was designed in the workbench. However, you needed to add your own training code to make the program functional. The next chapter will introduce some of the ways to train a neural network.