Simulated Annealing for Neural Networks

    Simulated annealing can also be applied to neural networks. This book provides a class named NeuralSimulatedAnnealing. By making use of the generic SimulatedAnnealing class introduced earlier in this chapter, this class implements a training solution for neural networks. Listing 7.1 shows a simple program that uses simulated annealing to train a neural network for the XOR operator.

Listing 7.1: Simulated Annealing and the XOR Operator (AnnealXOR.cs)

// Introduction to Neural Networks for C#, 2nd Edition
// Copyright 2008 by Heaton Research, Inc. 
// http://www.heatonresearch.com/online/introduction-neural-networks-cs-edi...
//         
// ISBN13: 978-1-60439-009-4  	 
// ISBN:   1-60439-009-3
//   
// This class is released under the:
// GNU Lesser General Public License (LGPL)
// http://www.gnu.org/copyleft/lesser.html

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using HeatonResearchNeural.Feedforward;
using HeatonResearchNeural.Feedforward.Train.Anneal;

namespace Chapter6XOR
{
    class AnnealXOR
    {

        public static double[][] XOR_INPUT ={
            new double[2] { 0.0, 0.0 },
            new double[2] { 1.0, 0.0 },
			new double[2] { 0.0, 1.0 },
            new double[2] { 1.0, 1.0 } };

        public static double[][] XOR_IDEAL = {                                              
            new double[1] { 0.0 }, 
            new double[1] { 1.0 }, 
            new double[1] { 1.0 }, 
            new double[1] { 0.0 } };

        static void Main(string[] args)
        {
            FeedforwardNetwork network = new FeedforwardNetwork();
            network.AddLayer(new FeedforwardLayer(2));
            network.AddLayer(new FeedforwardLayer(3));
            network.AddLayer(new FeedforwardLayer(1));
            network.Reset();

            // train the neural network
            NeuralSimulatedAnnealing train = new NeuralSimulatedAnnealing(
                    network, XOR_INPUT, XOR_IDEAL, 10, 2, 100);

            int epoch = 1;

            do
            {
                train.Iteration();
                Console.WriteLine("Epoch #" + epoch + " Error:" + train.Error);
                epoch++;
            } while ((epoch < 5000) && (train.Error > 0.001));

            network = train.Network;

            // test the neural network
            Console.WriteLine("Neural Network Results:");
            for (int i = 0; i < XOR_IDEAL.Length; i++)
            {
                double[] actual = network.ComputeOutputs(XOR_INPUT[i]);
                Console.WriteLine(XOR_INPUT[i][0] + "," + XOR_INPUT[i][1]
                        + ", actual=" + actual[0] + ",ideal=" + XOR_IDEAL[i][0]);
            }
        }
    }
}

    The NeuralSimulatedAnnealing class implements the Train interface, and thus can be used just like backpropagation and genetic algorithms discussed in earlier chapters. The NeuralSimulatedAnnealing is instantiated as follows:

NeuralSimulatedAnnealing train = new NeuralSimulatedAnnealing(
network, XOR_INPUT, XOR_IDEAL, 10, 2, 100);

    The NeuralSimulatedAnnealing class implements a special Randomize method. This method excites the state of the neural network in a way that is very similar to how the traveling salesman implementation works. The signature for the Randomize method is shown here:

override public void Randomize()

    First, MatrixCODEC is used to serialize the neural network into an array of double variables.

Random rand = new Random();
double[] array = MatrixCODEC.NetworkToArray(this.network);

    We then loop through the array.

for (int i = 0; i < array.Length; i++)
{

    Each array element is randomly excited based on the temperature.

  double add = 0.5 - (rand.NextDouble());
  add /= this.StartTemperature;

  add *= this.temperature;
  array[i] = array[i] + add;
}

    Finally, MatrixCODEC is used to turn the array back into a neural network.

MatrixCODEC.ArrayToNetwork(array, this.network);

    When this new neural network is returned to the Iteration function, it will be evaluated. If it is an improvement, it will be retained.