Neuron Layers

Get the entire book!

Neuron layers can also play an important part in the neural network's ability to learn. The neural network class presented in Chapter 3 allows you to define any number of different neuron layer types by overriding one or more of the methods of the neural network class. One such method is the threshold method. This method determines the activation function that will be used for the neural network. We will now examine some of the different neuron layer types you may wish to use. These types are summarized in Appendix E, “Neuron Layer Types”.

The Sigmoid Layer

A sigmoid neural network layer uses the sigmoid function to determine its activation. The sigmoid function is as follows.

The term sigmoid means curved in two directions like the letter “S”. You can see the sigmoid function in Figure 4.1. One important thing to note about the sigmoid layer is that it only positive values are returned. If you need negative numbers as a result from the neural network, the sigmoid function will be unsuitable. Most examples in this book will use the sigmoid layer if negative values are required, and the TanH layer if negative values are required.

Figure 4.1: The Sigmoid Layer

The TanhLayer Layer

As previously mentioned the sigmoid layer does return less than zero. It is possible to “move” the sigmoid layer to a region of the graph so that it now provides negative numbers. This is done using the TanhLayer. The equation for the TanhLayer activation function is given below.

Though this looks considerably more complex than the sigmoid function you can safely think of it as a positive and negative compatible version of the sigmoid function. The graph for the TanhLayer is shown in Figure 4.2.

Figure 4.2: The TanH Layer

The LinearLayer

The LinearLayer is essentially no layer at all. The linear layer does no modification on the pattern before outputting it. The function for the linear layer is given as follows.

The LinearLayer is useful in situations when you need the entire range of numbers to be outputted. Usually you will want to think of your neurons as active or non-active. Because the TanhLayer and SigmoidLayer both have established upper and lower bounds they tend to be used for more Boolean(on or off) type operations. The LinearLayer is useful for presenting a range. The graph of the linear layer is given in Figure 4.3.

Figure 4.3: The Linear Layer