Encog 2.0 and weight matrix

Submitted by whatswrong on Mon, 07/13/2009 - 12:03

Using Encog in Java

Hello everyone,

for a special task I've been using Encog 1.1 with a FeedForward network. I created a weightMatrix and initialised it with some needed values.

Then a selfmade method createNetwork() was invoked to create the network and set the pre defined weightMatrix.

With Encog 1.1 it all worked fine, the following snipped shows, how I did it:

Layer layerInput; Layer layerOutput; Network network; Matrix weightMatrix;

private void createNetwork() {
layerInput = new FeedforwardLayer(termCount);
layerOutput = new FeedforwardLayer(indexReader.numDocs());

network = new BasicNetwork();

layerInput.setMatrix(weightMatrix);

network.addLayer(layerInput); // Eingabeschicht (Terme)
network.addLayer(layerOutput); // Ausgabeschicht (Dokumente)
}

Now I tried to do the same with Encog 2.0 in two ways.

First attempt:

Synapse synapse; Layer layerInput; Layer layerOutput; Network network; Matrix weightMatrix;

private void createNetwork() {
layerInput = new BasicLayer(new ActivationSigmoid(), true, termCount);
layerOutput = new BasicLayer(new ActivationSigmoid(), true, indexReader.numDocs());

network = new BasicNetwork();

layerInput.setMatrix(weightMatrix);

network.addLayer(layerInput); // Eingabeschicht (Terme)
network.addLayer(layerOutput); // Ausgabeschicht (Dokumente)

network.addLayer(layerInput);
layerInput.addNext(layerOutput);
synapse = new WeightedSynapse(layerInput, layerOutput);
synapse.setMatrix(weightMatrix);
layerInput.addSynapse(synapse);
network.getStructure().finalizeStructure();
}

That results in an empty NeuralData object after network.compute(), so I can't use it.

Second attempt:

Synapse synapse; Layer layerInput; Layer layerOutput; Network network; Matrix weightMatrix;

private void createNetwork() {
layerInput = new BasicLayer(new ActivationSigmoid(), true, termCount);
layerOutput = new BasicLayer(new ActivationSigmoid(), true, indexReader.numDocs());

network = new BasicNetwork();

synapse = new WeightedSynapse(layerInput, layerOutput);
synapse.setMatrix(weightMatrix);
layerInput.addSynapse(synapse);
network.addLayer(layerInput);
network.addLayer(layerOutput);
network.getStructure().finalizeStructure();
}

Here it seems the predefined weightMatrix isn't assigned properly, there's no effect of using it.

So finally my question: How can I assign a weight matrix properly in Encog 2.0?

Submitted by jeffheaton on Tue, 07/14/2009 - 03:59.

In Encog 1.1 a layer could only connect to another layer, in a simple feed forward fashion. So this was a little easier. In Encog 2.0, each layer has a collection of next layers. As a result, the threshold values and weight matrix values have split. So to totally remember the state of the network and "hard code" it back in, you need to save the threshold values(from the layers), as well as the weight matrix (from the synapse).

I do something like this in the unit tests. I want the unit tests to ALWAYS start from the same "point", so I use a fixed starting weight matrix, rather than just setting them to random values. To do this, I use the NetworkCODEC, which can be used to reduce a network to an array of doubles, and then take it back to a network again. The following code (from the unit test( does this.

public static final Double[] RANDOM_NET = {-0.8289675647834567, 0.41428419615431555, -0.6631344291596013, -0.6347844053306126, 0.8725933251770621, 0.20730871363234438, 0.0693984428627592, 0.39495816342847045, 0.2876293823661842, -0.8091007635627903, 0.5170049536924719, -0.8775363794949156, 0.02786434379814584, -0.7373784461103059, 0.7670893161435932};

public static BasicNetwork createXORNetworkUntrained()
{
// random matrix data. However, it provides a constant starting point
// for the unit tests.

BasicNetwork network = new BasicNetwork();
network.addLayer(new BasicLayer(new ActivationSigmoid(),true,2));
network.addLayer(new BasicLayer(new ActivationSigmoid(),true,3));
network.addLayer(new BasicLayer(new ActivationSigmoid(),true,1));
network.getStructure().finalizeStructure();
NetworkCODEC.arrayToNetwork(RANDOM_NET, network);

return network;
}

I am going to work on a better example of doing this though. I should have an example where a network is built from scratch, with no training.

Submitted by jeffheaton on Tue, 07/14/2009 - 04:01.

I am going to create a complete example of how to build an XOR network from scratch and just set the weights and thresholds from hard coded values. But this might help you get started. This is how you build a three layer network with the layers and synapses. This way you have a local variable to for every component of the network.

// Build a network totally from scratch BasicNetwork network = new BasicNetwork();

// create three layers
Layer inputLayer = new BasicLayer(new ActivationSigmoid(), true,2);
Layer hiddenLayer = new BasicLayer(new ActivationSigmoid(), true,2);
Layer outputLayer = new BasicLayer(new ActivationSigmoid(), true,1);

// create synapses between those layers
Synapse synapseInputToHidden = new WeightedSynapse(inputLayer,hiddenLayer);
Synapse synapseHiddenToOutput = new WeightedSynapse(hiddenLayer,outputLayer);

// connect the synapses
inputLayer.getNext().add(synapseInputToHidden);
hiddenLayer.getNext().add(synapseHiddenToOutput);

// hook the layers to the network and finalize the structure
network.setInputLayer(inputLayer);
network.setOutputLayer(outputLayer);
network.getStructure().finalizeStructure();

network.reset();

Submitted by whatswrong on Tue, 07/14/2009 - 07:20.

Thank you very much! That solved my problem. Now, my method createNetwork is looking like that:
private void createNetwork() { network = new BasicNetwork();

layerInput = new BasicLayer(new ActivationSigmoid(), true, termCount);
layerOutput = new BasicLayer(new ActivationSigmoid(), true, indexReader.numDocs());

synapse = new WeightedSynapse(layerInput, layerOutput);
synapse.setMatrix(weightMatrix);

layerInput.getNext().add(synapse);

network.setInputLayer(layerInput);
network.setOutputLayer(layerOutput);

network.getStructure().finalizeStructure();
}