Simulated Annealing for Neural Networks

    Simulated annealing can also be applied to neural networks. This book provides a class named NeuralSimulatedAnnealing. By making use of the generic SimulatedAnnealing class introduced earlier in this chapter, this class implements a training solution for neural networks. Listing 7.1 shows a simple program that uses simulated annealing to train a neural network for the XOR operator.

Listing 7.1: Simulated Annealing and the XOR Operator

/**
 * Introduction to Neural Networks with Java, 2nd Edition
 * Copyright 2008 by Heaton Research, Inc. 
 * http://www.heatonresearch.com/books/java-neural-2/
 * 
 * ISBN13: 978-1-60439-008-7  	 
 * ISBN:   1-60439-008-5
 *   
 * This class is released under the:
 * GNU Lesser General Public License (LGPL)
 * http://www.gnu.org/copyleft/lesser.html
 */
package com.heatonresearch.book.introneuralnet.ch7.xor;

import com.heatonresearch.book.introneuralnet.neural.feedforward.FeedforwardLayer;
import com.heatonresearch.book.introneuralnet.neural.feedforward.FeedforwardNetwork;
import com.heatonresearch.book.introneuralnet.neural.feedforward.train.anneal.NeuralSimulatedAnnealing;

/**
 * Chapter 7: Training using Simulated Annealing
 * 
 * XOR: Learn the XOR pattern with a feedforward neural network that
 * uses simulated annealing.
 * 
 * @author Jeff Heaton
 * @version 2.1
 */
public class AnnealXOR {
	public static double XOR_INPUT[][] = { { 0.0, 0.0 }, { 1.0, 0.0 },
			{ 0.0, 1.0 }, { 1.0, 1.0 } };

	public static double XOR_IDEAL[][] = { { 0.0 }, { 1.0 }, { 1.0 }, { 0.0 } };

	public static void main(final String args[]) {
		final FeedforwardNetwork network = new FeedforwardNetwork();
		network.addLayer(new FeedforwardLayer(2));
		network.addLayer(new FeedforwardLayer(3));
		network.addLayer(new FeedforwardLayer(1));
		network.reset();

		// train the neural network
		final NeuralSimulatedAnnealing train = new NeuralSimulatedAnnealing(
				network, XOR_INPUT, XOR_IDEAL, 10, 2, 100);

		int epoch = 1;

		do {
			train.iteration();
			System.out
					.println("Epoch #" + epoch + " Error:" + train.getError());
			epoch++;
		} while ((epoch < 100) && (train.getError() > 0.001));

		// network = train.getNetwork();

		// test the neural network
		System.out.println("Neural Network Results:");
		for (int i = 0; i < XOR_IDEAL.length; i++) {
			final double actual[] = network.computeOutputs(XOR_INPUT[i]);
			System.out.println(XOR_INPUT[i][0] + "," + XOR_INPUT[i][1]
					+ ", actual=" + actual[0] + ",ideal=" + XOR_IDEAL[i][0]);
		}
	}
}

    The NeuralSimulatedAnnealing class implements the Train interface, and thus can be used just like backpropagation and genetic algorithms discussed in earlier chapters. The NeuralSimulatedAnnealing is instantiated as follows:

final NeuralSimulatedAnnealing train = new NeuralSimulatedAnnealing(
network, XOR_INPUT, XOR_IDEAL, 10, 2, 100);

    The NeuralSimulatedAnnealing class implements a special randomize method. This method excites the state of the neural network in a way that is very similar to how the traveling salesman implementation works. The signature for the randomize method is shown here:

public void randomize()

    First, MatrixCODEC is used to serialize the neural network into an array of Double variables.

final Double array[] = MatrixCODEC.networkToArray(this.network);

    We then loop through the array.

for (int i = 0; i < array.length; i++) {

    Each array element is randomly excited based on the temperature.

  double add = 0.5 - (Math.random());
  add /= getStartTemperature();
  add *= this.temperature;
  array[i] = array[i] + add;
}

    Finally, MatrixCODEC is used to turn the array back into a neural network.

MatrixCODEC.arrayToNetwork(array, this.network);

    When this new neural network is returned to the iteration function, it will be evaluated. If it is an improvement, it will be retained.