Local Minima and Randomization Options
Posted: Fri 19. Nov 2010, 20:52
Both Momentum and RPROP have to deal with not getting stuck in local minima. These methods do not have a lot of finesse in this area.
There are a number of things that can be done with randomization to jump out of local minima traps such as applying a dither reconnoiter when in the grasp of a local minima:
The following scripts could be added to Membrain:
___________________________________________________________
EDIT:
I cannot see any way to select weights with script. I am going to raise this potential issue to a new post.
There are a number of things that can be done with randomization to jump out of local minima traps such as applying a dither reconnoiter when in the grasp of a local minima:
- After significant training, save weights and explore current region in weight space looking for lower errors by using "shot gun" method:
- Check error from a "shot gun blast" centered at last known weight set (this was saved.)
- Perform for repeated shots and also for increasing blast radius.
- If a lower error is found, resume training at that location (use that particular weight perturbation)
- Distribution Type (Uniform, Gaussian, etc.)
- r, Interval or the amount of dither or "blast radius"
The following scripts could be added to Membrain:
- DitherWeights(Type, r)
- RandomizeType(Type)
- RandomizeInterval(r)
___________________________________________________________
EDIT:
I cannot see any way to select weights with script. I am going to raise this potential issue to a new post.