Escaping Local Minima with Genetic and Simulated Annealing Algorithms

Get the entire book!

In this chapter you will be shown how you can use simulated annealing and genetic algorithms to escape the local minima issues inherent with backpropagation. This chapter will focus primarily on the use of simulated annealing. Simulated annealing has become a very popular training algorithm, as it generally finds a better solution than backpropagation. We will begin by examining how simulated annealing will be used.

Using Simulated Annealing

As you recall from Chapter 9, simulated annealing works by simulating the cooling process of a metal. To apply this to a neural network we simply treat the weights and tolerances of the neural network as the individual ions in the metal. As the temperature falls, the weights of the neural network will achieve less excited states. As this process progresses the most optimal weight matrix is chosen, based on the error of the neural network. The error, as was discussed in chapter 3, is how well the neural network is recognizing the sample input. In the case of the example program in this chapter, that is the XOR problem.

Using Genetic Algorithms

Genetic algorithms seem to be less popular than simulated annealing for neural network training. Genetic algorithms take considerably more processing time, and memory than simulated annealing. This is because the genetic algorithm must keep a population of weight matrixes. Because of this thousands of weight matrixes must be kept through successive generations. Because of this chapter will focus on simulated annealing as a training method.

Nevertheless, this chapter does show an example of using a genetic algorithm. The genetic algorithm shown in this chapter is designed to work using the neural network class that was presented in Chapter 3. This algorithm is presented primarily for demonstrative purposes. It executes considerably slower than the simulated annealing example.