Introduction to Neural Networks for Java, Session 16

Session Material

(Final Exam Answers)

1. When would you use the hyperbolic activation function over the sigmoid activation function?

The hyperbolic activation function produces numbers in both the positive and negative range. Therefore, if you need numbers in this range the hyperbolic tangent activation function is a better choice than the sigmoidal activation function. If you expect positive only numbers, the sigmoidal function is acceptable.

2. Describe the structure the following network types: Hopfield, Self Organizing Map, Feedforward (2 hidden layers)?

Hopfield: a single recurrent layer. Each neuron is connected to the other neurons, but not to themselves. SOM: Input is first normalized and then sent to an input layer. From the input layer the output is directed to the output layer. The neuron with the highest activation is considered the winner and is the output. Feedforward: The input layer is connected to the first hidden layer, which is connected to the second hidden layer, which is then connected to the output layer.

3. Consider a feedforward neural network with 3 input neurons, 2 output neurons, and 2 hidden layers with 15 neurons each. How many matrixes, and what are their dimensions?

Three matrixes:

Input to Hidden-1: 3x15
Hidden-1 to Hidden-2: 16x15
Hidden-2 to Ouput: 16x2

4. You would like to design a predictive feedforward neural network to attempt to predict the daily temperatures. You would like to use the past ten daily temperatures to predict tomorrow's temperature. How many input neurons, and what would you feed into them? How many output neuron(s), and what would you expect from it/them?

10 Input neurons, would be given data corresponding to the ten temperatures preceding the day to be predicted.
1 Output neuron, would represent the day to be predicted.
The actual data presented to the neural network would likely be a percent as to what percent of the "maximum temperature" that day's temperature is.

5. What is simulated annealing based on? Briefly describe how it works.

Simulated annealing is based on the metallurgical process of annealing. It works by applying virtual heat that "excites" the weight matrixes of the neurons. The temperature is lowered and the weights fall into more logical patterns, in much the same way as slowly cooling a metal produces a more consistent and solid molecular pattern.

6. What happens if the backpropagation learning rate is too high? What happens if it is too low?

Too high of a learning rate will destabilize the network and not be able to bring the error rate to lower levels. Too low will train the network very slowly.

7. What is the difference between incremental and selective pruning.

Selective pruning starts with an already trained neural network and attempts to remove hidden neurons without substantially increasing the error rate for the neural network. Incremental pruning starts with a low number of neurons and creates new networks with larger numbers of hidden neurons until a suitable error level neural network is found.

8. You have a collection of insurance applications. You want to divide them into good, medium and low risk categories. Which type of neural network would you use? Why?

A self organizing map could be used to classify the applications into three groups. You could then analyze these three groups and see how closely they correspond with past performance of good, medium and low-risk applications.

9. Is it valid to prune input or output layer neurons? Why so or why not?

No it is not. Only hidden layer neurons should be pruned. Input and output neurons define the fundamental nature of the problem you are trying to solve. Pruning is only an optimization technique, it should not change the nature of the problem. Pruning is about finding the optimal level of hidden layers and neurons.

10. What is the purpose of momentum in backpropagation? What does it seek to solve?

Momentum is a percent value. It determines what percent of the pervious learning iteration's deltas should be applied to the current iteration. This can help to get the training past local minima.

Videos for this Session

Video	Title
	Introduction to Neural Networks for Java(Class 16/16)