Summary
In this chapter we saw how simulated annealing and genetic algorithms can be applied used to help backpropagation escape from a local minimum. A local minimum is a dip in the error function that back propagation can easily become stuck on. By not evaluating beyond the crest of the local minimum the backpropagation algorithm loses on a chance to find other minima that may be more suited than the current local minimum. The lowest point that the error curve will ever reach is called the global minimum. The optimal training process will find the global minimum.
Simulated annealing can be used to further optimize a neural network. To use the simulated annealing algorithm the weights must be aligned in a linear structure such as an array. Then the simulated annealing process begins by “randomizing” these weight values taking into consideration the current “temperature” and the suitability of the current weight matrix. The temperature is decreased and the weight matrix ideally converges on an ideal solution. This process continues until the temperature reaches zero or no improvements have occurs for a specifies number of cycles. The simulated annealing algorithm executes relatively quickly.
Likewise, genetic algorithms can be used to further optimize a neural network. To use a genetic algorithm the weights must be aligned in a linear structure such as an array. This array will be used as a chromosome. Then the genetic algorithm proceeds to splice the genes of this weight matrix with other suitable weight matrixes. Through subsequent generations the suitability of the neural network should increase as less fit weight matrixes are replaced with better suited ones. This process continues until no improvements have occurred for a specified number of generations. The genetic algorithm generally takes up a great deal of memory and executes much slower than simulated annealing. Because of this simulated annealing has become a popular method of neural network training.
Of course the process of simulated annealing and genetic algorithms may produce a less suitable weight matrix than what was started with. This can happen when a simulated annealing or a genetic algorithm is used against an already trained network. This lack of improvement is not always a bad thing, as the weight matrix may have moved beyond the local minimum. Further backpropagation training may allow the neural network to now converge on a better solution. However, it is still always best to remember the previous local minimum incase a better solution simply cannot be found.
This chapter showed you how to use simulated annealing and backpropagation to produce a weight matrix that is better suited to the problem the neural network is being trained for. This process only changes the weights of the synaptic connections. In the next chapter we will see how we can further optimize to change the structure of the neural network. Chapter 11 will introduce the topic of pruning, which allows us to remove synaptic connections that do not contribute to the output of the neural network. This results in neural network that will take less time to execute.
