Local Minima and Randomization Options

Want to provide feedback to MemBrain or post/discuss suggestions for improvements or new features? Detected a bug that should be resolved in a future version?

Then this is the right place for your contribution!
Post Reply
MrCreosote
Posts: 55
Joined: Wed 21. Jul 2010, 18:43

Local Minima and Randomization Options

Post by MrCreosote »

Both Momentum and RPROP have to deal with not getting stuck in local minima. These methods do not have a lot of finesse in this area.

There are a number of things that can be done with randomization to jump out of local minima traps such as applying a dither reconnoiter when in the grasp of a local minima:
  • After significant training, save weights and explore current region in weight space looking for lower errors by using "shot gun" method:
  • Check error from a "shot gun blast" centered at last known weight set (this was saved.)
  • Perform for repeated shots and also for increasing blast radius.
  • If a lower error is found, resume training at that location (use that particular weight perturbation)
These shot gun blasts are simply applying dither to the saved weights. This dither is a random number distribution from [1-r, 1+r] which is used to multiply the saved weights. It is defined by the parameters:
  • Distribution Type (Uniform, Gaussian, etc.)
  • r, Interval or the amount of dither or "blast radius"
In addition, Randomize could be given some control with regard to the same parameters: Distribution Type, and Interval.

The following scripts could be added to Membrain:
  • DitherWeights(Type, r)
  • RandomizeType(Type)
  • RandomizeInterval(r)
Custom Script is another method - especially since one has been provided to manually assign the Normalization Min and Max Limits to the SNeuralProp object. I am basically writing a Script to scale Initial Weights produced by Randomize. Since there is a script Random number generator, I can build uniform and normal distributions so I should be able to do everything I mentioned above via script. Still, I think some of the above would be convenient. NOTE: I think Type is nowhere near as important as Interval - especially when in this case, the objective is to demonstrate sensitivity to Initial Weights.

___________________________________________________________

EDIT:

I cannot see any way to select weights with script. I am going to raise this potential issue to a new post.
User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Local Minima and Randomization Options

Post by Admin »

Hi,

many thanks for the input, sounds like a good suggestion.

I'm currently in the process of finalizing the configurable and alkso scriptable Net Error function and will have another look at this afterwards.

Regards,
Thomas
Thomas Jetter
User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Local Minima and Randomization Options

Post by Admin »

A first version of the 'Shotgun' feature now has been implemented in MemBrain 03.07.00.00 that just went online.

Regards
Thomas Jetter
Post Reply