difference between calculateError & getError, Bug?
I created a Backpropagation object(bp) for training a feedforward MLP with one hidden layer. basicnetwork is trained until bp.getError() was less than 0.02 .
After training stopped, when i use basicnetwork.calculateError function for see error on training set, give me: 1.152316933510082
why there is different? I set stopping training at less 0.02 error but when i test the data set used for training get me 1.152316933510082 !
Is there a bug in Encog?
Please help me. Thanks.
RSS
YouTube
Twitter
Facebook
They will retrun a different value
But not generally a huge value. The reason for this is as follows.
Propagation training works by calculating the error. To calculate the error I must iterate over every training set element, which takes time. So I don't want to do this any more than I have to.
To calculate the training adjustment I need to calculate the error for each training set element and as I do this I also calculate the overall error. Then, once I've processed the entire training set, I modify the weights in such a way that usually produces a better result(backpropagation). The problem is, the error(which is reported by train.getError) is from my loop where I was calculating the differences, BEFORE the adjustment is made.
When you go to call basicnetwork.calculateError, it is working on the modified neural network, so it usually should be slightly better than the .getError reported by the last training iteration.
To test this I modified the XOR backprop example to display the final training iteration error, as well as the final error of the network, as reported by basicNetwork.getError.
I get the following results:
Epoch #3924 Error:0.009999918617581394
Final network error:0.009998509740975152
If its a really big difference, there is something else going on. Make sure its the same training set as the network was trained with.
Jeff
I thinks there is a bug in calculating error?
I am sure that i use same training set as the network trained with.
But This is Wonderful! I set maximum error to 0.01. When training stopped, I test training set but output of network is wrong, very very wrong?! This makes me surprised! May be in calculating error in Encog exists bug. If want, i can send my source code.
Is it possible my network was wrong? network is MLP(feedforward) with hidden layer, i set all activation function to ActivationSigmoid(), Thredhold parameter for layers is true, training method is Backpropagation.
I think there is a bug in Encog. If want, i can send my source code.
okay, try this...
I still can't reproduce this. This is also an area I use quite a bit, since I train networks and then turn right around and use them.
Use the following, code for your training loop.
int epoch = 1;
do {
train.iteration();
System.out.println("Epoch #" + epoch + " Error:" + Format.formatPercent(train.getError()));
System.out.println("Evaluated error: " + Format.formatPercent(train.getNetwork().calculateError(trainingSet)));
epoch++;
} while(train.getError() > 0.01);
System.out.println("Final evaluated error: " + Format.formatPercent(train.getNetwork().calculateError(trainingSet)));
The code above displays the error reported by the training function at each loop iteration and then the second shows the "evaluated error" just after the training loop is done. Notice they are not exactly the same, which is normal.
They actually move together in pars, as each Epoc's evaluated error is the same as the next epoc's training error. Try this technique with your data and see how close they stay together.
Epoch #1 Error:56.997184%
Evaluated error: 56.997184%
Epoch #2 Error:56.997184%
Evaluated error: 52.843587%
Epoch #3 Error:52.843587%
Evaluated error: 50.120023%
Epoch #4 Error:50.120023%
Evaluated error: 51.324022%
Epoch #5 Error:51.324022%
Evaluated error: 51.296656%
Epoch #6 Error:51.296656%
Evaluated error: 51.270567%
Epoch #7 Error:51.270567%
Evaluated error: 50.072441%
Epoch #8 Error:50.072441%
Evaluated error: 50.634920%
Epoch #9 Error:50.634920%
Evaluated error: 50.627715%
Epoch #10 Error:50.627715%
Evaluated error: 50.614936%
...
Epoch #68 Error:4.308266%
Evaluated error: 2.846251%
Epoch #69 Error:2.846251%
Evaluated error: 1.700173%
Epoch #70 Error:1.700173%
Evaluated error: 0.883071%
Epoch #71 Error:0.883071%
Evaluated error: 0.391167%
Final evaluated error: 0.391167%
Neural Network Results:
0.0,0.0, actual=2.1195601254130874E-5,ideal=0.0
1.0,0.0, actual=0.9961354019551435,ideal=1.0
0.0,1.0, actual=0.995249927137771,ideal=1.0
1.0,1.0, actual=0.004868884513333364,ideal=0.0
Result
This is result. Only for running faster, i changed 0.01 to 0.02
run:
Epoch #1 Error:2.392933%
Evaluated error: 113.232199%
Epoch #2 Error:1.769253%
Evaluated error: 104.223779%
Final evaluated error: 104.223779%
As you see, there is large different between train.getError() and network.calculateError()!
Also outputs of network have large different with ideal outputs(i don't insert outputs top, if want i can do it)!
Okay my last idea. Are you
Okay my last idea. Are you using a ContextLayer? If so, it may be holding context and be out of sync between training and error evaluation. If that is the case try calling network.clearContext() just before you call calculateError.
Failing that, I would almost need to see source code to verify if it is an Encog bug.
What is your mail address?
Please give me a email address to send source code to you or send me a mail to getting it:
[[email edited]]
okay sent you an email
I also edited your email address from the post so it won't be out in the public.
yes this is a bug
I analyzed the code and found that if you specify the number of threads to be greater than the number of training set elements you will get an "incorrect" and very low error rate. Generally it is best to set the thread count to zero(the default) and allow Encog to choose the number of threads to use, based on your processor count and training set size.
This issue has been fixed in the beta version of 2.4, and has been checked in.
Thanks for the bug report!