Hello, when can it be possible for MemBrain to have a feature allowing the net to be "reset to zero" at an arbitrarily parameter pre-set interval in the middle of the core processes (that being the forward-pass where the gradients sequential-order and overall-structure are being pre-determined and prepared for the incoming back-pass that updates the weights and activation thresholds) of the teach step?
As a side comment just to express how hopeful I am that I can expect it to be possible this request can be fulfilled.
I was recently I began to really understand the beauty and meaning behind these two features: "Repetitions per Pattern" and "Re-Apply Pattern (when repeating Patterns)"... I began to see that you were probably making an other way to manage time variant net designs to be able to carry out their functionalities. for example: if "Repetitions per Pattern" parameter is set to greater than 1 repetitions while "Re-Apply Pattern..." is set to off, then this carry out a time skip function in the forward pass part of the teaching prorcess were delayed links can get time to skip and catch up to another designated timing.
To see that these things were being considered in how Membrain was deliberately designed rekindled and revitalized my admiration and awe for the software.
Feature to reset the net multiple times (at arbitrary interval) during the teaching process, not only at the lesson end
-
learner411
- Posts: 7
- Joined: Sun 21. May 2023, 02:42
Re: Feature to reset the net multiple times (at arbitrary interval) during the teaching process, not only at the lesson
Hallo,
I'm not sure if I really and fully understand your request, I will give it anoter try, however.
As per your feedback to "Repetitions per Pattern" and "Re-Apply Pattern (when repeating Patterns)":
The feature gets important only in case one or more of the the input neurons has an "Activation Sustain Factor" < 1. In this case, the activation of these neurons will decay with every "Think Step" that is performed during teaching since the Teacher will not re-apply the input data for each repeated pattern execution.
In case the "Activation Sustain Factor" is 1 (default for input neurons), the "Re-Apply" setting has no effect, since the input neuron will keep its last applied activation (sustained forever until another activation is set).
--> You can give this a try by creating an input neuron and setting the Activation Sustain Factor to e.g. 0.9. Then click on "Think Step" repeatedly and you will see that the activation decays.
The feature can be interesting for instance like this:
- You have binary (1 and 0) input data for training (e.g. an array of pixels in an image)
- Your net shall, however, not only react to binary inputs but also to weaker input levels.
- With the "Repeat Pattern" feature and the "Not Re-Apply" feature you can achieve that the net will also see "weaker" patterns than just "value 1" in its input neurons during training. You have to set the "Activation Sustain Factor" of the inputs to < 1 for this, however.
I'm not sure if I really and fully understand your request, I will give it anoter try, however.
As per your feedback to "Repetitions per Pattern" and "Re-Apply Pattern (when repeating Patterns)":
The feature gets important only in case one or more of the the input neurons has an "Activation Sustain Factor" < 1. In this case, the activation of these neurons will decay with every "Think Step" that is performed during teaching since the Teacher will not re-apply the input data for each repeated pattern execution.
In case the "Activation Sustain Factor" is 1 (default for input neurons), the "Re-Apply" setting has no effect, since the input neuron will keep its last applied activation (sustained forever until another activation is set).
--> You can give this a try by creating an input neuron and setting the Activation Sustain Factor to e.g. 0.9. Then click on "Think Step" repeatedly and you will see that the activation decays.
The feature can be interesting for instance like this:
- You have binary (1 and 0) input data for training (e.g. an array of pixels in an image)
- Your net shall, however, not only react to binary inputs but also to weaker input levels.
- With the "Repeat Pattern" feature and the "Not Re-Apply" feature you can achieve that the net will also see "weaker" patterns than just "value 1" in its input neurons during training. You have to set the "Activation Sustain Factor" of the inputs to < 1 for this, however.
Thomas Jetter
Re: Feature to reset the net multiple times (at arbitrary interval) during the teaching process, not only at the lesson
Hi, the feature is implemented now, just went online with new MemBrain version 15.3.0.0.
- Attachments
-
- ResetNetAfterNPatterns.jpg
- Screen shot of new feature in Teacher Editor
- (44.31 KiB) Not downloaded yet
Thomas Jetter
-
learner411
- Posts: 7
- Joined: Sun 21. May 2023, 02:42
Re: Feature to reset the net multiple times (at arbitrary interval) during the teaching process, not only at the lesson
I think you nailed it perfectly as far as I can tell. Beautiful. I believe I have even proven it to myself by running inference on a trained net and see that it does obey a cutoff length in accordance to being specified. This is a game changer by there way where my use for it is concerned. I could try to tell you how thankful and exited I am, but let me just drop the effort and hope that by now you get it needless to say. This is soooooo great to have. This Software Tool is ever evolving and still how its beginner user friendly charm. Cheers! to what the future holds, so worth looking forward to.
Yours Sincerely,
learner411.
Yours Sincerely,
learner411.