Are some companies easier to predict than others?

I am beginning some research into stock market prediction. This will serve two primary goals.

First, I want Encog to really "flex its muscles" in terms of training on some large input data. Soon enough Encog 2.1 will begin and I really hope to increase training efficiency, as well as introduce concurrency that should really speed up Encog on dual, quad and dual-quad machines. Grid is a goal as well.

Secondly, I want to create quite a few examples for financial forecasting using Encog 2.x. This article series is the beginning of this.

I decided to start simple. I am using an input window of 10 days of trading to predict the next day (day 11). I trained the neural network with 10 years worth of market data using only the stock price. In the future I want to experiment with using sectors, where, other stocks in the same sector may be used as indicators for yet other stocks in that sector. More on that later.

For this experiment I wanted to determine two things. First, what is the optimal number of hidden layers. Encog 2.0 has a brute force method for attempting to determine optimal hidden layer structure. You input max/min hidden layers and counts into the pruning algorithm and then it simply trains every combination and sees which one has the best error drop-off. Using 500 iterations of RPROP training, over the ten years I was able to determine that 29 neurons in the first hidden layer and 13 in the second worked well. It took nearly 2 days of processing to do this on a dual core machine. The pruning algorithm even in Encog 2.0 is fully threaded and takes advantage of multicore.

Then using this feedforward neural network I wanted to see if certain members of the S&P 500 were easier to predict than others. Again, using ONLY the historical price of the stock as an indicator. To do this I collected ten years of data on each of the S&P stocks and trained each for 1000 iterations of RPROP. This was quite a few trianing sets given that it was 10 years of data. This took almost 4 days of training on my dual core. Again, using threaded code. All Encog 2.0 stuff.

You can see the results here:

http://www.heatonresearch.com/node/1020

Some stocks performed quite a bit better than others. Stocks such as CBS and UPS seemed to train much more easy than stocks on the other end of the spectrum, such as Dell. I have no explanation for this, other than that some stocks seem to exhibit more "rhythmic" price fluctuation than others.

This is just the beginning of this article series, and more will be coming as I continue the research. Part of the delay will be training times as it can take consierable time to train these large datasets.

Jeff