Both Momentum and RPROP have to deal with not getting stuck in local minima. These methods do not have a lot of finesse in this area.
There are a number of things that can be done with randomization to jump out of local minima traps such as applying a dither reconnoiter when in the grasp of a local minima:
- After significant training, save weights and explore current region in weight space looking for lower errors by using "shot gun" method:
- Check error from a "shot gun blast" centered at last known weight set (this was saved.)
- Perform for repeated shots and also for increasing blast radius.
- If a lower error is found, resume training at that location (use that particular weight perturbation)
These shot gun blasts are simply applying dither to the saved weights. This dither is a random number distribution from [1-r, 1+r] which is used to multiply the saved weights. It is defined by the parameters:
- Distribution Type (Uniform, Gaussian, etc.)
- r, Interval or the amount of dither or "blast radius"
In addition, Randomize could be given some control with regard to the same parameters: Distribution Type, and Interval.
The following scripts could be added to Membrain:
- DitherWeights(Type, r)
Custom Script is another method - especially since one has been provided to manually assign the Normalization Min and Max Limits to the SNeuralProp object. I am basically writing a Script to scale Initial Weights produced by Randomize. Since there is a script Random number generator, I can build uniform and normal distributions so I should be able to do everything I mentioned above via script. Still, I think some of the above would be convenient. NOTE: I think Type is nowhere near as important as Interval - especially when in this case, the objective is to demonstrate sensitivity to Initial Weights.
I cannot see any way to select weights with script. I am going to raise this potential issue to a new post.