Page 1 of 1

Local Minima and Randomization Options

Posted: Fri 19. Nov 2010, 20:52
by MrCreosote
Both Momentum and RPROP have to deal with not getting stuck in local minima. These methods do not have a lot of finesse in this area.

There are a number of things that can be done with randomization to jump out of local minima traps such as applying a dither reconnoiter when in the grasp of a local minima:
  • After significant training, save weights and explore current region in weight space looking for lower errors by using "shot gun" method:
  • Check error from a "shot gun blast" centered at last known weight set (this was saved.)
  • Perform for repeated shots and also for increasing blast radius.
  • If a lower error is found, resume training at that location (use that particular weight perturbation)
These shot gun blasts are simply applying dither to the saved weights. This dither is a random number distribution from [1-r, 1+r] which is used to multiply the saved weights. It is defined by the parameters:
  • Distribution Type (Uniform, Gaussian, etc.)
  • r, Interval or the amount of dither or "blast radius"
In addition, Randomize could be given some control with regard to the same parameters: Distribution Type, and Interval.

The following scripts could be added to Membrain:
  • DitherWeights(Type, r)
  • RandomizeType(Type)
  • RandomizeInterval(r)
Custom Script is another method - especially since one has been provided to manually assign the Normalization Min and Max Limits to the SNeuralProp object. I am basically writing a Script to scale Initial Weights produced by Randomize. Since there is a script Random number generator, I can build uniform and normal distributions so I should be able to do everything I mentioned above via script. Still, I think some of the above would be convenient. NOTE: I think Type is nowhere near as important as Interval - especially when in this case, the objective is to demonstrate sensitivity to Initial Weights.



I cannot see any way to select weights with script. I am going to raise this potential issue to a new post.

Re: Local Minima and Randomization Options

Posted: Thu 25. Nov 2010, 06:21
by Admin

many thanks for the input, sounds like a good suggestion.

I'm currently in the process of finalizing the configurable and alkso scriptable Net Error function and will have another look at this afterwards.


Re: Local Minima and Randomization Options

Posted: Sun 5. Dec 2010, 16:54
by Admin
A first version of the 'Shotgun' feature now has been implemented in MemBrain that just went online.