Unexpected error "Trying to assign invalud number to matrix"

Submitted by kylepolich on Thu, 04/02/2009 - 21:21

Using Encog in Java

Hi,

I am using the IntroNeuralNetworks.jar file in my Java application in Eclipse. I build and train my NN and everything seems okay. However, when I call computeOutputs, the follow exception is thrown:

com.heatonresearch.book.introneuralnet.neural.exception.MatrixError: Trying to assign invalud number to matrix: NaN

I checked the input array that I am passing as a parameter. It is the correct size and none of its values are NaN, only valid double values.

Help!

Submitted by soupbone on Fri, 04/03/2009 - 10:51.

Is that you are attempting to put a number into a matrix that is either a NaN or Infinity. If you post the complete stacktrace from the error I can get more info.

When I usually see this is during training. If you use Backprop and you set your learning rate too high, then the network becomes really unstable. And as a result of the deltas being applied too strongly(from too high of a learning rate or momentum) some of the values can get adjusted outside of Java's valid range for numbers.

You might try a lower learning rate while training. If that doesn't help, post a stack trace and we can take a look.

Submitted by kylepolich on Fri, 04/03/2009 - 19:39.

Thank you for your reply. I had learning rate set to .001 and momentum to .1. I multiplied both by .1 to try and test the theory that these rates are too high, but alas, no improvement. Below is a stack trace, although to my eyes it is not particularly insightful.

Thanks in advance,

Kyle

com.heatonresearch.book.introneuralnet.neural.exception.MatrixError: Trying to assign invalud number to matrix: NaN

at com.heatonresearch.book.introneuralnet.neural.matrix.Matrix.set(Unknown Source)
at com.heatonresearch.book.introneuralnet.neural.feedforward.FeedforwardLayer.createInputMatrix(Unknown Source)
at com.heatonresearch.book.introneuralnet.neural.feedforward.FeedforwardLayer.computeOutputs(Unknown Source)
at com.heatonresearch.book.introneuralnet.neural.feedforward.FeedforwardNetwork.computeOutputs(Unknown Source)
at com.heatonresearch.book.introneuralnet.neural.feedforward.train.backpropagation.Backpropagation.iteration(Unknown Source)
at neuralNetTest.NeuralNet.trainBackprop(NeuralNet.java:93)
at neuralNetTest.NeuralNet.setTrainingSets(NeuralNet.java:62)
at neuralNetTest.NeuralNetTest.main(NeuralNetTest.java:13)

Submitted by jeffheaton on Sat, 04/04/2009 - 08:47.

Yes, that is right in the middle of the backprop learning. This is not unusual with backprop. Often on complex sets I get that, and just keep lowering the learning rate until it goes away. Encog will detect and handle this better, but with the released version it is something of a manual process. On larger or complex sets I've sometimes used learning rates as low as 0.0000001. The bigger the set the more deltas are accumulated in the batch, so as the set gets bigger the learning rate must fall.

Another option is to use a training method other than bprop. Annealing might work well. This is also where resilient propagation (RPROP) is very useful, but its not in the released version, also coming in 2.0. I am actually working on the RPROP stuff today.

Jeff

Submitted by kylepolich on Sat, 04/04/2009 - 16:02.

Thank you for your suggestion. Unfortunately this was only partially applicable to my particular problem, Turning down the learning rate alone did not fix the problem. However, (although I'm not sure what motivated me to do so), I normalized my dataset to [-1, 1] from its previously larger domain. Once I did this (and turned the learning rate down) - bada-bing! Suddenly the net starts working. As I'm just getting started with NNs, I cannot say why the normalization was important yet, but it did the trick.

In general, how should one determine an appropriate learning rate and momentum?

Submitted by jeffheaton on Sun, 04/05/2009 - 18:41.

I've put code into Encog 2.0 that should make it less likely to get that particular error. But I am glad you got your network to train.

Setting the learning rate and momentum can be tricky. And unfortunately, the most optimal values vary as the network is trained. For Encog 2.0 I provide training strategies that you can add to automatically adjust the learning rate and momentum as learning progresses. We also have resilient propagation, which requires no parameters and is largely automatic. However, it will be at least a month before Encog 2.0 is released in a beta form.

So until then, here is what I do. I make the program so that it will save the neural network after training, so that I can resume training. This allows me to adjust the parameters as need be. I start with a low momentum. Sometimes NO momentum. And then I start with a training rate of 0.1. If it trains just fine, I may try bumping it up to get training to go faster. If the error rate just goes crazy, or I get the invalid number error, I decrease. Usually to 0.1 then 0.01 then 0.001 until the network is learning better.

Momentum is useful to escape a local minimum. You've hit one of those when your network appears stuck and is no longer moving the error rate down, and the rate is NOT where you would like it. It could be that that is just the best level that the network will reach, and that is "just the way it is". But this is also a case where bumping up the momentum can help, and push it over the local minimum.

Making this automatic in Encog 2.0 works relatively well. RPROP works so much better because it basically has a learning rate separate for each weight. And then the learning rate changes independently for each weight to try and maximize learning. It works really well, and will be one of the major features of Encog 2.0.

Jeff

Submitted by sergio on Tue, 04/07/2009 - 00:55.

I've look through all examples, but I don't understand how can I save neural network. I tried to do this using serialization, but I get java.io.NotSerializableException on java.util.logging.Logger. Then I tried to use EncogPersistedCollection, but get NullPointerException, because createPersistor() method in BasicNetwork return null.
What shoud I do for saving the net on the disk? (I use encog core r421)

Submitted by jeffheaton on Tue, 04/07/2009 - 18:18.

You are checking it directly out of SVN. This code is not yet stable, I am in the process. Encog 2.0 is getting there, but the persistence piece is still broken. I would suggest trying Encog 1.1. It is still about a month before I will have all the kinks worked out of 2.0.

http://code.google.com/p/encog-java/downloads/list

I did not know about the logger issue with serialization, thanks for finding that! :)

Jeff

Submitted by sergio on Fri, 04/10/2009 - 11:10.

Thanks for you previous answer!
Saving in Encog 1.1 works excellent.
What would I say about subject... Bug in org.encog.neural.data.csv.CSVNeuralDataSet, in method public void close() you write for (final CSVNeuralIterator iterator : this.iterators) and this is right, but you never add elements in this.iterators... and this isn't right :-)

In method public Iterator iterator() you simply return new CSVNeuralIterator(), if write something like this :
CSVNeuralIterator iter = new CSVNeuralIterator(); this.iterators.add(iter); return iter;

All will be works fine!
Fix this please in version 2.0 :-)
P.S. Sorry for my english...