Using the JOONE Editor

Submitted by jeffheaton on Sun, 12/23/2007 - 21:33

Get the entire book!

Introduction to Neural Networks with Java

JOONE is an open source 100% Java
implementation. JOONE is released under a Limited GNU Public License (LPGL).
What this means is that you are free to use JOONE in any project, be it
commercial or private, without paying any sort of royalty or licensing free. The
complete source code to JOONE is contained on the companion CD-ROM that is
included with this book. The CD-ROM also contains the compiled binaries that are
used to actually include JOONE as part of your project.

JOONE is maintained at the Source Forge
(http://www.sourceforge.com). Source Forge is a site that hosts files, free of
charge, for open source projects. If you would like more information about JOONE
you should visit the JOONE project page at the Source Forge (http://joone.sourceforge.net/).
Because JOONE is always being updated to include new features it is a good idea
to check the site to see if a new version of JOONE exists. This book uses the
latest version of JOONE that was available at the time of printing, which is
version 0.9.0 of the engine and version 0.6.0 of the editor. JOONE contains
several components. To begin learning to use JOONE you will first be introduced
to the JOONE editor.

JOONE includes a neural network editor. This
allows you to graphically layout neural networks. This allows you to quickly see
the components of a neural network and how they fit together. The editor allows
you to create neural network solutions that do not require any customized
coding. If your solution can be accomplished with only a neural network and no
other supporting code, then you can use only the JOONE editor.

Of course few programming tasks can be
accomplished with only a neural network. You will often have to write support
code that encapsulates the neural network and provides the necessary
infrastructure, such as the user interface, that most programs require.

Because of this limitation, most applications
that use JOONE make use of the JOONE engine. The JOONE engine is the set of core
neural network classes that you can use directly from within your own Java
programs. What the JOONE editor is primarily used for is to quickly try out
several neural network architectures before deciding which is best for your
program. The JOONE editor allows you to see your neural networks in a very
graphical way.

Setting Up Your Environment

Before you can begin using JOONE editor or
engine you must make sure that the Java environment is correctly setup on your
computer, and that JOONE is installed. The instructions for setting up your
environment differ greatly depending on what sort of operating system you are
running. If you are running a Windows based operating system, refer to Appendix
C, "Compiling Examples Using Windows". If you are running on a UNIX or Linux
based system refer to Appendix D, "Compiling Examples Using UNIX/Linux". Once
you have your environment properly setup and JOONE installed you can begin using
JOONE.

JOONE consists of several components. At the
heart of everything is the JOONE engine. The JOONE engine has no user interface
and implements all of the low level functions of the network. If you write
programs that make use of the JOONE system, these programs will most likely make
use of the engine. JOONE also contains a GUI editor that can be used to quickly
design and test neural networks. Figure 3.1 shows a UML diagram of some of the
main JOONE Engine components.

Figure 3.1: JOONE Components

Regardless of if you are using the JOONE Editor
or the JOONE Engine you will be using these components. These components are
listed briefly here. As you build the neural networks in this chapter you will
see how many of these objects are actually used.

FileInputSymapse A synapse that is designed to
input from a disk file.
FileOutputSynapse A synapse that is designed to
output to a disk file.
InputSynapse A interface that defines a input
synapse, or data source.
InputTokenizer A tokenizer us uses to transform
underlying input formats into arrays of floating point numbers that will be
fed to the neural network.
Layer The layer object is the basic element in JOONE. A
layer transfers the input pattern to the output pattern while executing a
transfer function. The output pattern is then sent to all output synapses that
are attached to the layer.
LinerLayer A neuron layer that uses a simple
linear function to activate
Monitor The Monitor is used to control the
behavior of the neural net. It controls the start/stop actions and permit to
set the parameters of the net. Each layer and synapse is connected to a
monitor.
PatternTokenizer A interface that is used to
define a tokenizer.
SigmoidLayer neuron layer that uses a sigmoid
function to activate.
SimpleLayer This abstract class represents layers
that are made up of neurons that implement some transfer function.
StreamInputSynapse This abstract class forms the
base for any stream based input synapses.
StreamOutputSynapse This abstract class forms the
base for any stream based output synapses.
Synapse The Synapse is the connection element
between two Layer objects. Each connection between neurons in these layers is
represented by a weight. The values of these weights are the memory of the
neural network.
TanhLayer A neuron layer that uses the hyperbolic
tangent to activate
URLInputSynapse A synapse that is designed to
input from a HTTP connection.

To see how to use JOONE objects an example is
in order. You will now be shown how to use the JOONE Editor and the JOONE
Engine. Our discussion of JOONE will begin with the JOONE editor.

Solving the XOR Problem with the JOONE Editor

The examples given in this chapter will revisit
the XOR problem. First you will see how to use the JOONE editor to create a
neural network that is capable of solving the XOR problem. In addition to
modeling the neural network, the JOONE editor also allows you to run training
data against the neural network. This allows the neural network to learn, by
adjusting neuron weights, to solve the XOR problem. This XOR solution neural
network will first be created in the JOONE editor so that you can visually see
what the neural network looks like.

Once you have seen how to use the JOONE editor
to solve the XOR problem, you will be shown how to use the JOONE engine to
create a standalone Java application that solves the XOR problem.

The JOONE editor is used to
quickly model neural networks and observe the output. The JOONE editor will not
generate code, nor is it ever used in conjunction with any program you might
create. The JOONE editor is simply a rapid prototype tool that can be used to
visually see neural networks in action. In this chapter you will be stepped
through the process of creating a simple multi-layer feed forward neural
network.

This section shows you how to lay
out a neural network using the editor. The next section of this chapter will
show you how to take that model and turn it into a working program. You will
begin with a very relatively simple problem; creating a neural network that can
learn the XOR problem.

You will recall from Chapter 1,
"Introduction to AI" that the XOR problem confounded the perceptron. In this
chapter we will use the JOONE Editor to create a neural network that is capable
of solving the XOR problem. The XOR function is summarized in Table 3.1.

Table 3.1: The XOR Function

Input 1	Input 2	Output
0	0	0
0	1	1
1	0	1
1	1	0

To have JOONE solve the XOR problem, as
presented in Table 3.1, the XOR problem must be presented to JOONE in a manor
that JOONE can process. JOONE requires that all such input be in a standard data
file format.

Understanding JOONE Data Files

JOONE is programmed to only perform input and
output from files. This is true for both the JOONE Editor, as well as the JOONE
Engine. There are three types of data files that you will likely process with
JOONE. They are summarized as follows.

Training Files are used to provide JOONE with
sample inputs to train with. If you are using supervised learning this file
will also contain the anticipated output from the neural network.
Input Files are used to provide input that the
neural network should process. Once the neural network has been trained, the
network can be fed input files. This will produce result files.
Result Files contain the results from the output
neurons. A result file is generated when the neural network is fed input
files.

To have JOONE solve the XOR problem you must
create the appropriate data files. The format for all three files is exactly the
same. Just by looking at the file contents of a JOONE data file it is impossible
to discern one file from another.

A JOONE data file consists of individual lines
of floating point numbers which are delineated by semicolons. Decimal points are
not required, if the numbers are integer. The individual numbers can be either
input values or the expected result. This will be communicated to JOONE when
these files are specified. When a file is provided as input you tell the JOONE
Editor or Engine which columns are input and which columns are expected output.
The training file that would be created to train the XOR network is shown in
Listing 3.1.

Listing 3.1: The XOR Function Training File (training.txt)

0.0;0.0;0.0

0.0;1.0;1.0

1.0;0.0;1.0

1.0;1.0;0.0

The training file shown in Listing 3.1 provides
the neural network with all anticipated neural network inputs. It is not
necessary, and usually impossible, to provide JOONE with every possible
combination of inputs. However, for a simple function like XOR it is easy enough
to provide every combination. This is the purpose of the first two columns of
Listing 3.1. The remaining column specifies the output for each of the input
pairs. For example, the input zero followed by one should produce an output of
one.

These input files are the only way to
communicate with the JOONE Editor or Engine. For batch programs this limitation
is fine. But for more complex programs, particularly real-time interactive
applications, this is something of a limitation presented by JOONE. Solving this
limitation will be the last topic of this chapter. By creating a few extra
classes to extend the JOONE Engine you can directly feed data to JOONE without
the use of a file.

Understanding JOONE Layer File

Neural networks are composed of layers of a
neurons. A layer consists of one or more neurons that have similar properties.
JOONE provides you with several different layer types. To allow the editor to
easily be expanded, a special configuration file is used to identify what layer
types are supported by JOONE.

When you launch the JOONE editor you must
specify the location of the layers.xml file. A copy of layers.xml is included on
the companion CD-ROM along with this JOONE editor. For more information on the
proper installation of the JOONE Editor and Engine, refer to Appendix C and D.

This chapter will make user of only one layer
type, the Sigmoid layer. Chapter 4, "How a Machine Learns" will introduce the
other layer types supported by JOONE. Chapter 4 will also show how JOONE can
easily be extended to support additional neuron layer types. Appendix E, "Neuron
Layer Types" provides a reference of the neuron layer types and their
properties. With the JOONE Editor and your layers file properly installed you
are now ready to construct a neural network.

Constructing the Neural Network

To begin creating your neural network you must
first launch the JOONE editor. This requires that your environment be properly
setup. The section "Setting up Your Environment" presented earlier in this
chapter shows how to do this. To launch the JOONE editor you should issue the
following command from a command or terminal window.

java
org.joone.edit.JoonEdit c:\jar\layers.xml

This command can be executed from any location
on your computer. The first parameter, which specifies the location of your
layers file, may be different depending on where you have your layers file
stored. Refer to the previous section for more information about the layers
file.

When the editor first comes up a blank neural
network will be presented. You can add components to this network and then later
save. The companion CD-ROM already has a copy of this neural network saved under
the name xor.ser.

To create the XOR solution neural network you
must add three sigmoid layers. Individual neurons are not shown on JOONE
diagrams, only neuron layers. These three layers will be the input, hidden and
output layers.

First create the input layer. This is done by
clicking on the sigmoid layer button on the toolbar. As you hover your mouse
over each of the buttons the meaning of the button is shown in the status bar
that is at the very bottom of the JOONE editor. On some operating systems popup
will also be shown to tell you what component your mouse is currently hovering
over. If this is done correctly your screen should look like figure 3.2.

Figure 3.2: A Sigmoid layer

You must now set the properties for the input
layer. Clicking on the input layer will display the properties dialog box. You
should rename this layer to "Input Layer" and set the rows parameter to two.
This dialog box is shown in Figure 3.3. The current version of JOONE contains no
"Save" or "Apply" button in JOONE property dialog boxes, your changes are saved
when you click the close button for a window. There is no way to discard
changes.

The JOONE editor will open the properties
window for any object that you click. This can be an annoying side effect, as
you will get the properties window opened even when you simply select a object
to move it. A more elegant solution is on the current request list for the JOONE
project. The best way to counter this is to simply leave the properties window
open all the time, and allow it to change to whatever the currently selected
object is. Of course, you should move the properties window away from your
current work area so it does not interfere with your work.

Figure 3.3: Sigmoid layer properties

As you can see there are several
properties that a sigmoid layer allows you to set. These properties remain the
same regardless of if you are creating a neural network in the JOONE Editor, or
if you are directly programming the JOONE Engine from Java. The meanings of each
of these fields are summarized below.

Layer Name This is the layer name, its value is not
directly used by JOONE. When it becomes necessary to debug a program, a layer
name can be invaluable for determining exactly which layer a particular Java
reference is pointing to.

Rows The rows parameter specifies the number of
neurons in this level.

Next the hidden layer must be created. The
steps to create a hidden layer are nearly the same as the steps to create the
input layer. Create another sigmoid layer and set its name to "Hidden Layer" and
the rows parameter to three.

Next you must create the output layer. The
steps to create the output layer are nearly the same as the steps to create the
previous layers. Create a third sigmoid layer and set its name to "Output Layer"
and the rows parameter to one.

You now have three disjoint neural layers. For
these layers to function they must be linked together. Begin by connecting the
input layer to the hidden layer. This is done by dragging a line from the small
circle on the right hand side of the input layer. You must have the input layer
selected to see this circle. This should draw an arrow. JOONE will likely open
up the properties window when you complete this operation. Simply close the
properties window as you do not need to make any changes. You should also draw a
line from the hidden to the output layer in exactly the same way. At this point
your network is complete. This is the point at which the startXor.ser sample was
saved that is included on the companion CD-ROM. Your neural network should now
look like Figure 3.4.

Figure 3.4: Sigmoid layer properties

Now the neural network must be trained.
However, before the network can be trained, some additional objects must be
added to the neural network. First a "File Input Layer" must be added to the
left of the input layer. This layer can be obtained from the toolbar, just like
the sigmoid layer. The small circle from the file input layer should be dragged
to the input layer. This is done in the same way as when the other layers are
connected. This will cause the training file to feed its data to the input
layer.

You must also set the properties for the file
input layer. To bring up the properties, click the input layer. Set the firstCol
and lastCol parameters to one and two respectively. This specifies which columns
specify the input data in the data file. The filename should contain the
complete path to your train.xml file. This file can be found on the companion
CD-ROM. All other parameters should remain the same. Figure 3.5 shows a
completed file input layer dialog box.

Figure 3.5: A completed file input layer dialog box

As you can see there are several
properties that a file input layer allows you to set. The meanings of each of
these fields are summarized below.

Buffered Should JOONE buffer the values or read them
from disk each time.

Enabled Is this layer enabled.

FileName The name of the file that JOONE should read
from. Must be in the format shown earlier in Listing 3.1.

FirstCol One input file can serve multiple purposes.
This parameter specifies the starting column to use. Columns indexes begin with
one.

FirstRow This parameter specifies the starting row.
Set to one to specify the first row.

IgnoreBefore Specifies the number of input patterns
to ignore for each training cycle. I have not found much practical use for this
property.

LastCol This is the last column index that this
layer should read.

LastRow This is the last row that the layer should
read from the file. To instruct the layer to read the entire contents of the
file a value of zero should be passed.

Name The name of this layer.

StepCounter If this property is set to true, then
the file will automatically rewind when the end is reached. Because most neural
networks are rapidly trained with the same data, this is often desired.

Next a teacher layer must be added to the
output later. This will allow the weights for the input, hidden and output
layers to be modified to cause the neural network to learn the XOR pattern. Add
a teacher layer to the right of the output layer. Connect the output layer to
the teacher layer by dragging the small circle from the output layer to the
teacher layer. The only property of the teacher layer that should be set is the
name. The name property of the teacher layer should be set to "Teacher Layer".

The teacher layer requires access to the input
data too. The teacher layer will be concerned with the last column of Listing
3.1, which is the anticipated output. A second file input layer should be added
to the neural network. This file input layer should be named "Teacher File
Input", as it will be connected to the teacher layer. The properties of the
teacher file input layer are set as follows. The input file to the teacher file
input layer is the same input file as was specified for the main "File Input
Layer". The teacher layer uses the desired output column of Listing 3.1; as a
result both the firstCol and lastCol properties must be set to three.

The "Teacher File Input Layer" should be
connected to the "Teacher Layer". This is done by dragging the small box on the
edge of the "Teacher File Input Layer" to the "Teacher Layer". With these steps
complete your neural network is now ready to begin learning. The completed
neural network should look like Figure 3.6.

Figure 3.6: A neural network ready for training

1.5.1.
Training the Neural Network

Now that the neural network has been
constructed the training process can begin. Select the "Control Panel" menu.
From here select the "Control Panel" menu item. They are both named "Control
Panel". This will display a window that allows you to specify the training
parameters. These parameters should be filled in as seen in Figure 3.7. The
totCycles parameter should be set to 10,000. The patterns parameter should be
set to 4. The learning rate parameter should be set to 0.8 and the momentum
parameter to 0.3. Finally, to specify that you are training you should set the
learning property to true.

Figure 3.7: Setting training parameters

Learning Set this value to true if you are training
the neural network. If you are presenting patters to the neural network for
recognition this property should be set to false.

Learning Rate The learning rate for the neural
network controls the rate at which changes are made to the weights store in the
synapses. This will be covered in greater detail in Chapter 4.

Momentum A parameter used for back propagation
learning. The use of momentum will be covered in Chapter 9, "Understanding
Backpropagation".

Input Patterns The number of input patterns that
will be presented to the neural network.

Pre-learning Prelearning allows several cycles to be
designation as learning cycles, even though the learning property is set to
false.

Total Cycles How many times the input pattern should
be run. This is usually used with training to determine how long to train for.

To begin the training process clicks the start
button. JOONE is completely multi-threaded and other tasks can be performed
while the network is training. The neural network will be taken through 100,000
training cycles. For very large or complex neural networks this can take
considerable time. For such tasks, JOONE can operate in a distributed mode,
using many computers to train the neural network.

At the end of the training the error, specified
as RMSE (root mean square error), should be very small. An error below 0.1 is
acceptable. If this fails to happen you must retrain the neural network. To do
this select the "Randomize" menu item under the "Control Panel" menu. Now train
the neural network again. You may wish to save your neural network now that it
is trained.

Now that the neural network has been trained we
can present data and observe the results. The neural network should now be
properly trained to respond with the correct XOR result for any given input. The
neural networks objects must be rearranged slightly to allow the neural network
to be executed. This will be discussed in the next section.

1.5.2.
Preparing to Execute the Neural Network

Before the neural network can be tested, a few
changes must be made to the neural network diagram. The teacher must be
disconnected from the neural network and a file output layer must be added. Add
a file output layer to the right of the output layer. Click on the new file
output layer and enter a file name to store the results to. The companion CD-ROM
contains a sample results file that is named results.txt. You should also name
this layer "Results".

The "Results" layer must be attached to the
output layer in place of the teacher layer. To do this click the line that
connects the output layer to the teacher layer. Now press the delete button to
sever this link. Next drag a new link from the "Output Layer" to the newly
created "Results" layer. Your model should now look like Figure 3.8.

Figure 3.8: A neural network ready to run

Now that the neural network has been properly
configured to be ran you can see if the neural network properly learned the XOR
function. Select the "Control Panel" menu. From here select the "Control Panel"
menu item. They are both named "Control Panel". This will display a window that
allows you to specify the neural network parameters. This is the same window as
was used to train the neural network. These parameters should be filled in as
seen in Figure 3.9.The totCycles parameter should be set to 1. The patterns
parameter should be set to 4. Finally, to specify that you are no longer
training you should set the learning property to false.

Once these fields are properly set you should
click the "Begin" button. The neural network will run very quickly. The file
results.txt will be created. What has happened is that every input pattern
specified in the training file train.txt, discussed earlier, has been run
through the neural network. The result of each row is written to a corresponding
row in the result.txt file. The contents of this file are shown in Listing 3.2.

Figure 3.9: Ready to run the neural network

Listing 3.2: Neural network results (result.txt)

0.012022188543571214

0.9850650966922243

0.9848523804373928

0.018901538819852792

Your results may vary from Listing 3.2.
However, you should always see two numbers which are close to one, and two
numbers which are close to zero. This shows that the neural network has recalled
one for the second and fourth rows and zero for the first and third.

Using the JOONE Engine

As you saw in the previous section it is
possible to construct a neural network using the JOONE Editor. With the neural
network complete you can both train and execute neural networks using input
files. While the JOONE Editor is user friendly and makes it easy to edit neural
network layouts, it does not lend itself well to more complex programming tasks.

Generally neural networks do not stand alone as
independent programs. A neural network is almost always integrated into a much
larger program. The neural network is responsible for performing pattern
recognition, and the larger program handles the user interface. To use JOONE
with a larger Java program you must use the JOONE Engine, rather than the JOONE
Editor. The JOONE Engine gives you a library of classes that you can use to
implement neural networks.

The JOONE engine is written to be very
multi-threaded. On computer systems with many processors, or in a distributed
environment, JOONE can achieve great performance. Unfortunately these additional
layers mean a great deal of overhead, and programming complexity for otherwise
simple projects. In this book we will not use the JOONE Engine. In the next
section we will examine a reusable class named "Network.java" that we will use
in later chapters as well to implement the feedforward backpropagation neural
network. In this chapter we will focus mainly on how to use it. Later chapters
will explain how it process information, as well as how it learns.