Page 1 of 1

Raw CSV input

Posted: Tue 13. Jan 2015, 10:26
by giselle007
Hello everybody,
I have a problem with the input of raw CSV files. I have a computer with american settings and somewhere on the forum I read this:
"The CSV files which come with this packet have been generated on a machine with German operating system. As such, they use the field limiter 'semocolon' (';') and the decimal limiter 'comma' (',').
Should the script be used on a machine with anglo-american operating system setting then the characters in the CSV files must be replaced accordingly:
First, replace all ',' by '.' - then replace all ';' durch ','""
I did this and now my a piece data looks like:

In1;In2;In3;In4;In5;In6;In7;In8;In9;In10;In11;In12;In13;In14;In15;In16;In17;In18;In19;Out
0;0;0;0;0;0;0;0;0;0;-0,000279252;0;0;0;0;0;-0,000445236;0;0;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
-0,000399627;0;0;0;0;0;0;0;0;-0,000465983;0;0;0;0,002351558;0;0;0;0;-0,005064317;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0,000294031;0;0;0;0;0;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0,007551791;0
0;0;0;-0,0002292;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0,00059312;0;0;0;0
-0,003186828;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0
0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0

I have 19 input neurons with the same name as the first line and one output neuron.
If i try to import this I always get the error: "mismatch between I/O definition and patterns".
If I put only the output in a seperate file, than he does accept it, but I can't get him to accept the input...
Does anybody see where it goes wrong?

Re: Raw CSV input

Posted: Tue 13. Jan 2015, 11:36
by TJetter
Hello,

the data you posted still is formatted according to the German CSV separators: Columns are separated by ';' and decimal separator is ','.
For an English OS machine you need to separate the data columns with ',' and the decimal separator must be '.' as described in the post you reference.
Could it be that you posted the data from the original file, prior to your adaptations?

Once you have the correct CSV file you should try the following:
- Open your net with 19 inputs and one output
- In the Lesson Editor click on 'Names from Net'
--> This will align the column layout in the Lesson Editor with your net
Alternatively you can adjust the number of columns for inputs and outputs in the Lesson Editor manually to 19 and 1, respectively.
- In The Lesson Editor menu select 'Raw CSV files' --> 'Import Current Lesson (Raw CSV)
Now you should have the data in the Lesson Editor.

Once you have the data imported you can save the data as a MemBrain Lesson (Lesson Editor menu 'Lesson Files'). This file can be loaded in the lesson Editor at any time without having to configure the columns previously, since this file contains all information with respect to the data layout.

Regards

Re: Raw CSV input

Posted: Tue 13. Jan 2015, 11:53
by giselle007
In1,In2,In3,In4,In5,In6,In7,In8,In9,In10,In11,In12,In13,In14,In15,In16,In17,In18,In19,Out
0,0,0,0,0,0,0,0,0,0,-0.000279252,0,0,0,0,0,-0.000445236,0,0,0
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-0.000399627,0,0,0,0,0,0,0,0,-0.000465983,0,0,0,0.002351558,0,0,0,0,-0.005064317,0
0,0,0,0,0,0,0,0,0,0,0,0,0,0.000294031,0,0,0,0,0,0
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.007551791,0
0,0,0,-0.0002292,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.00059312,0,0,0,0
-0.003186828,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0


This is how the first lines of my data look now. And the first lines are similar to the names of the neurons in the network.
And I followed all your steps, but it still just gives the error:"mismatch between I/O definition and patterns".
If i make a model with one input neuron and one output neuron and i just give values for that one neuron than he does accept it ... So it must be something with the delimiter...

Re: Raw CSV input

Posted: Tue 13. Jan 2015, 18:31
by TJetter
Try the following:
- Click 'Names from Net' in the Lesson Editor
- Add two or so patterns manually in the Lesson Editor (There's a button to add new patterns)
- Enter some values in some of the columns, use '.' as decimal separator
- Export the lesson to Raw CSV via the Lesson Editor
- Open the file in a text editor
--> Now you have an example of what your file has to look like

If you still can't find out what the problem is then post your net and CSV file here. Also post the example file exported from MemBrain as described above.

Regards

Re: Raw CSV input

Posted: Tue 13. Jan 2015, 22:09
by giselle007
I split my data up in smaller pieces and now it works.
Is there a certain limit to the amount of parameters?

Re: Raw CSV input

Posted: Wed 14. Jan 2015, 06:50
by TJetter
There is some limitation for the CSV file import, it should be several 10.000 lines or so but depends on the actual amount of data.
If the CSV file is too large you can split it up like you did, then same the single imported lessons in the MemBrain format (Lesson files, *.mbn).
Once you have imported and saved all lessons this way you can combine them in the Lesson Editor using the 'Append...' command in its File Menu.
Then save the integrated large lesson as one mbl file.

This also has the advantage that the mbl files load several times faster than CSV files.

Regards

Re: Raw CSV input

Posted: Tue 20. Jan 2015, 06:53
by TJetter
CSV import capabilities have been significantly improved with respect to import speed and allowed file size in the recent version of MemBrain (06.00.04.00).
You may want to give it a try again with your full data size files now.

Regards