Cascade Correlation with Hidden Node Networks

Have specific questions about how to work with certain MemBrain features? Not sure about which checkbox in MemBrain has which effects? Want to know if a certain functionality is available in MemBrain or not?

Your questions will be appreciated here!
Post Reply
MrCreosote
Posts: 55
Joined: Wed 21. Jul 2010, 18:43

Cascade Correlation with Hidden Node Networks

Post by MrCreosote »

Just to see what would happen, I tried CC on a custom built network with hidden nodes and connections which were not full layer to layer.

First a quick description of my custom network:
  • Imagine two 3-layer networks (In,Hid,Out) where they are combined, in parallel, to produce a single output: the output nodes of the two networks would become another hidden layer and the new output node would be an additional, 4th layer.
  • The theory is that if inputs come from 2 different classes, allow the nnet to basically do a reduction in dimensionality before they are combined netwise to get an output.
What I thought I saw was that CC would basically set the output of the Hiddens to ZERO and then start working at turning them on, one at a time. (This is a guess on my part since I was not able to make sense out of the process.)

If that is true, however, that would be nice because one could build an architecture with an abundance of Hidden nodes, connected as desired, and CC would simply arrive at some solution where a number of these nodes would be unused.

QUESTION: Is this how CC would operate on such a network?
User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Cascade Correlation with Hidden Node Networks

Post by Admin »

Hi Tom,

won't have the time today to reply to this in more depth. Would suggest that you download the new MemBrain version that has the bug fix on the Cascade Correlation teachers (just went online) and give it a try first. This might simplify things, i.e. discussing about real observations is easier than talking about 'what would happen' ;)

Regards,
Thomas
Thomas Jetter
MrCreosote
Posts: 55
Joined: Wed 21. Jul 2010, 18:43

Re: Cascade Correlation with Hidden Node Networks

Post by MrCreosote »

Explored behavior a bit and it looks like this probably isn't the way to do things.

What appears to happen is that newly generated nodes connect everything together: Inputs, Hiddens, Outputs.

The vast majority of original hiddens had values of 0 so it seems they were removed so to speak.

I believe I've seen enough to conclude that this is not the way to do things with CC.

Thanks
Tom
User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Cascade Correlation with Hidden Node Networks

Post by Admin »

There's one thing you could try additionally:

Neurons in MemBrain have a property flag 'Allow Teacher to Connect Output'. If you deactivate this flag for the neurons you don't want to receive new output connections then the CC teacher won't do so.

Don't know if this helps with your approach, though.

Regards,
Thomas
Thomas Jetter
Post Reply