Mackey Glass time series prediction

You work on a certain topic or data set and don't know how to start off with it using MemBrain? Not sure if your net design matches your problem or if there is room for optimization? Is it reasonable at all to approach your problem with NNs? Is MemBrain the correct tool to accomplish your task and to match your infrastructure?

These questions are best placed here!
dinufira
Posts: 7
Joined: Thu 7. Oct 2010, 13:41

Mackey Glass time series prediction

Post by dinufira » Thu 21. Oct 2010, 09:00

Hello,

I am new in neural nets in general and in Membrain in particular. I red the Beginner's Tutorial and I have to solve some problems, but it's not easy for me to find the steps and their order for giving the net all the necessary data. In short, the situation is as follows :
I have several time series that contain 830 to 950 numbers. I had them in Excel .xls files that I have transformed in Excel .csv files. Every time serie is a column that contains the numbers. I want to predict the next number. I will upload an attachment the contains the first time serie in .csv format.
I have downloaded the examples from your site and let's say that I have chosen the net named MackeyGlassTime Variant 1. I have the training, validation and number detection files.
Please give me your support for introducing correctly the input data for 1 time serie and make the net understand what I want (the a/m prediction).
I need your assistance for the steps I have to follow and also to understand exactly what I am doing. Your assistance learning how to solve the problems will be highly appreciated.
Thanks in advance for your understanding and support.

Dinu Fira

User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Mackey Glass time series prediction

Post by Admin » Thu 21. Oct 2010, 18:09

Dear Dinu,

first, let me say that time series prediction is the most difficult thing to start with when it comes to NN since then you have to deal with time variant networks where the output does not only depend on the input but also on the history of the net (i.e. the calculation steps it already has performed since it was reset the last time).

There is one simple solution to get around a time variant network: You can equip the network with more than one input, i.e. one input for every past value. Let's assume you want to predict the next value based on the four previous ones then you could create a net with four inputs (t, t-1, t-2, t-3) and one output (t+1). This is still a time invariant network although you use it with data that represents a time series.
The disadvantage of this approach is that the net can only leverage from the past 4 values in order to predict the next one and that there is more efforts required for preparing the data accordingly. Still, this can be a good thing since it is much easier to handle!

However, for the moment let's assume you want to use a real time variant network for your solution.

In this case I have automized the most critical steps for you through the script file I have attached. You need to adjust some settings in the script file in order to match your needs. However, you may want to give it a try first as it is:

- Extract the attached file, it will create a separate folder.
- In MemBrain select 'Scripting' - Execute Script' and select the file 'TimeSeries.as' from the just created folder.

Here is what the script does:
- Load a training and a validation lesson
- Train five different net candidates with these data for a specified training time and determine the best candidate
- Merge training and validation data into one single new lesson that has all data points from the start up to the present
- Re-train the best net candidate with this combined lesson
- Record the final output of the net on this lesson plus an adjustable number of future extrapolated data points into a CSV file.

The data files (Train.mbl and Validate.mbl) in the example are taken from a sine wave. The script is adjusted to extrapolate 4 points of the sine wave into the future.
After the script has executed, open the file 'Extrapolated.csv' in Excel, add a column to the left in order to have an X-Axis. Then you can easily create a plot of the input and output data to the net over the whole time series, including the four extrapolated data points in the very end.

To perform the same with your data you need to do the following:

- Prepare at least one net with one input x(t) and one output x(t+1).
- Next, you need a training and a validation data set. Each of them needs to have two columns:
- X(t)
- X(t+1)

It is very important that the validation data makes up only a few data points and the majority of the data goes into the training lesson. Furthermore, the validation data points must take over seamlessly where the training data end, i.e. the validation represents the valid appendix of the training time series up to the last known data point in time.

In MemBrain, load your net, open the lesson editor and click 'Names from Net'. This sets up your lesson editor with the same I/O number and names as the net. Then, in the lesson editor select 'Raw CSV files' - 'Import Current Lesson (RAW CSV)'. Select the training csv file which is then loaded. In the Lesson Editor, select 'Lesson Files' - 'Save Current Lesson As...' and save the training data as 'Train.mbl'.
Do the same for the validation data, save it as 'Validate.mbl'.

Next, you need to ensure that the nets you want to use are adjusted to the correct Normalization Range:
Open each net candidate, select both the INPUT and the OUTPUT neuron, press ENTER, click on 'Normalization', enable the check box 'Use Normalization' and enter your specific valid data minimum and maximum limits according to the range of your data. Leave some headroom for future data!

- Copy the files you have created into the directory where the script resides
- In the script 'TimeSeries.as' adjust the following according to your needs:
-- NUMBER_OF_NETS
-- NET_FILE_NAME_BASE
-- EXTRAPOLATE_COUNT
-- LIST_SEPARATOR
-- DEC_SEPARATOR

Note that all your net candidates must have a name that comprises of the value assigned to NET_FILE_NAME_BASE plus a number that starts at '1' for the first net candidate.

Please note that most of the steps above are one-time steps. The onl ything you need to do for every data set is the creation of a training and a validation data set.
Instead of using the names 'Train.mbl' and 'Validate.mbl' you can also use different names. You can adjust the names in the script file instead:
- TRAIN_LESSON_NAME
- VALIDATION_LESSON_NAME

If you want to learn what happens in the script in details try to read through the code. Even if you are not a programmer you should be able to get an idea of what is going on according to the comments in the script and the names of the functions that are called. Watch out for the function 'void main()', this is the main program and entry point of the script.
When reading MemBrain script files then best use an Editor with syntax highlighting for C/C++ or Java. The scripting example download on the MemBrain homepage includes a description of how the free editor PSPad can be set up to best display MemBrain script files and even to support debugging them.

Note that one of the called functions actually is not located in the file 'TimeSeries.as' but in the file 'TrainAndExtrapolate.as'. This has historical reasons, these were separate scripts in the past. You could also cut/paste the function into the actual executable script file ''TimeSeries.as' and delete the second script file.

As you see there is quite a lot to do to handle time variant networks properly, that's why doing this manually is both tedious and error prone and I would have to write tons of pages to get this explained.

Hope that gives you a good start!

Regards
TimeSeriesPrediction.zip
Script that trains and validates several net variants for time series prediction, selects the best one and predicts an adjustable number of future values. Demo data contained in the sample is a sine wave.
(10.61 KiB) Downloaded 718 times
Thomas Jetter

dinufira
Posts: 7
Joined: Thu 7. Oct 2010, 13:41

Re: Mackey Glass time series prediction

Post by dinufira » Fri 22. Oct 2010, 10:46

Dear Thomas,

Thank you very much for your reply, as well for all your advices. I will start as you indicated and even if it's a lot of work I am determined to achieve the goal.
When I will get stuck I will appeal again to your help.

Thanks again and best regards.

Dinu

dinufira
Posts: 7
Joined: Thu 7. Oct 2010, 13:41

Re: Mackey Glass time series prediction

Post by dinufira » Thu 13. Jan 2011, 13:48

Dear Thomas,

First of all I wish you a Happy New Year and much success in your work.
I had to delay for 2 months the beginning of my "training" with the lesson you posted so kindly for me as I had a very crowded period.
I made a first attempt as per your instructions, but I had a few problems, as follows :
- after executing the script an error message appeared (just the word "Error" in a separate window of the one with Script Trace)
- the best trained net was no. 5
- I have opened the file Extrapolated.csv and added a column to the left. If I want to save this file with the column added I have some warning messages from Excel that I might lose some informations if I keep this format. I have to mention that I use on my computer Excel 2003 (with which I also created the Raw.csv files, that are not all accepted by Membrain (some are some not) and I wonder why) as I don't like at all Excel 2007. The Raw.csv files were intially .xls files and I saved them with Excel 2003 as .csv files. Is it wrong ?
- I have skipped the action "prepare at least one net (??) with one input x(t) and one output x(t+1) and the training and validating data set, as I didn't understand exactly what to do. Can you help me here ?
- I loaded in Membrain the best trained net and I followed your instructions until "Do the same for the validation data, save it as "validate.mbl" where I got stuck because I don't know how to do it.
Then I preffered to delete all and to try again from the beginning. Therefore I have deleted the whole folder Time Series Prediction, I downloded your zip file and I started all again. Now the best trained candidate was no.4, but the same error message appeared in the separate window. Now I am at the same point after I added the column to the left in the Extrapolated .csv. I am now disoriented and I need your assistance in this fog I am.
Sorry to bother you but as I don't have experience with NN I couldn't understand very well what and how to do.
Hope that you don't mind and that you will find some time to teach me more.

Thanks in advance for your understanding and support.

Yours truly,

Dinu Fira

User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Mackey Glass time series prediction

Post by Admin » Thu 13. Jan 2011, 20:36

Dear Dinufira,

Happy New Year for you, too!
dinufira wrote:- after executing the script an error message appeared (just the word "Error" in a separate window of the one with Script Trace)
We first have to resolve this one.
Do you mean that right when the zip file has been extracted and you execute the script you get this message box "Error"? Before you made any modifications to the files that have been extracted?
What MemBrain Version are you using (Help-->About)?

Also, what Windows Version are you running? Did you extract the folder in a location where admin rights are required? This may render the script unable to create files on the disk for example.

Regards,
Thomas
Thomas Jetter

dinufira
Posts: 7
Joined: Thu 7. Oct 2010, 13:41

Re: Mackey Glass time series prediction

Post by dinufira » Fri 14. Jan 2011, 15:21

Dear Thomas,

The message Error in a separate window appers after the script is executed and before any modification to the extracted files.
I use Membrain version V03.06.00.00 (Jan 30 2010).
I am running original and licensed Windows 7 Professional.
I extracted the file in an area that no administrator rights are required. Membrain creates in the folder TimeSeriesPrediction the file Extrapolated.csv so I don't think that is a problem. Anyhow my login to the computer is always with administrator rights.

Thank you for your promptitude and time.

What's next ?

Regards,

Dinu

Dear Dinufira,

Happy New Year for you, too!
dinufira wrote:- after executing the script an error message appeared (just the word "Error" in a separate window of the one with Script Trace)
We first have to resolve this one.
Do you mean that right when the zip file has been extracted and you execute the script you get this message box "Error"? Before you made any modifications to the files that have been extracted?
What MemBrain Version are you using (Help-->About)?

Also, what Windows Version are you running? Did you extract the folder in a location where admin rights are required? This may render the script unable to create files on the disk for example.

Regards,
Thomas[/quote]

User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Mackey Glass time series prediction

Post by Admin » Fri 14. Jan 2011, 18:09

Hi Dinufira,
dinufira wrote:I use Membrain version V03.06.00.00 (Jan 30 2010).
Would you mind upgrading to 03.07.00.00 which is the latest one available?

I just want to be sure that you work with the same and latest version and I want to see if that Error goes away.

For further steps I need to dive a little bit deeper into your descriptions and questions, won't have time today, though.

I'll get back to you as soon as I find the time to.

Regards,
Thomas
Thomas Jetter

dinufira
Posts: 7
Joined: Thu 7. Oct 2010, 13:41

Re: Mackey Glass time series prediction

Post by dinufira » Wed 26. Jan 2011, 10:47

Hi Thomas,

I followed your advice and upgraded to version V03.07.00.00 (Dec 5 2010), I run the script and after it finishes there is no more any Error message. That's good news as the Error went away, as per your intention.
The best trained candidate was again Net 5. I have saved the Net5 Trained and I wait for your further instructions.
In order to save time I tried to upload 4 time series that I want to make predictions to but I get back the message "The extension csv is not allowed". As previously mentioned, the files were saved in .csv format with Excel 2003. How can I check if these files are suitable for Membrain ?
Awaiting your reply,

Best regards.

Dinu

Admin wrote:Hi Dinufira,
dinufira wrote:I use Membrain version V03.06.00.00 (Jan 30 2010).
Would you mind upgrading to 03.07.00.00 which is the latest one available?

I just want to be sure that you work with the same and latest version and I want to see if that Error goes away.

For further steps I need to dive a little bit deeper into your descriptions and questions, won't have time today, though.

I'll get back to you as soon as I find the time to.

Regards,
Thomas

dinufira
Posts: 7
Joined: Thu 7. Oct 2010, 13:41

Re: Mackey Glass time series prediction

Post by dinufira » Wed 26. Jan 2011, 11:18

Hi again, Thomas,

On second thought I have zipped the 4 time series files and I uloaded them. Can you check if these files are OK for Membrain ?
Thanks and best regards.

Dinu
dinufira wrote:Hi Thomas,

I followed your advice and upgraded to version V03.07.00.00 (Dec 5 2010), I run the script and after it finishes there is no more any Error message. That's good news as the Error went away, as per your intention.
The best trained candidate was again Net 5. I have saved the Net5 Trained and I wait for your further instructions.
In order to save time I tried to upload 4 time series that I want to make predictions to but I get back the message "The extension csv is not allowed". As previously mentioned, the files were saved in .csv format with Excel 2003. How can I check if these files are suitable for Membrain ?
Awaiting your reply,

Best regards.

Dinu

Admin wrote:Hi Dinufira,
dinufira wrote:I use Membrain version V03.06.00.00 (Jan 30 2010).
Would you mind upgrading to 03.07.00.00 which is the latest one available?

I just want to be sure that you work with the same and latest version and I want to see if that Error goes away.

For further steps I need to dive a little bit deeper into your descriptions and questions, won't have time today, though.

I'll get back to you as soon as I find the time to.

Regards,
Thomas
Attachments
69.zip
(4.18 KiB) Downloaded 609 times

User avatar
Admin
Site Admin
Posts: 438
Joined: Sun 16. Nov 2008, 18:21

Re: Mackey Glass time series prediction

Post by Admin » Sat 29. Jan 2011, 14:56

Hi Dinufira,

good that the script now runs without any problems.

The next step in my original instructions is:
Admin wrote:To perform the same with your data you need to do the following:

- Prepare at least one net with one input x(t) and one output x(t+1).
- Next, you need a training and a validation data set. Each of them needs to have two columns:
- X(t)
- X(t+1)

It is very important that the validation data makes up only a few data points and the majority of the data goes into the training lesson. Furthermore, the validation data points must take over seamlessly where the training data end, i.e. the validation represents the valid appendix of the training time series up to the last known data point in time.
I think this is where you are stuck right now, correct?

The CSV files you posted only have ONE column. If you read my instructions above you see that you need TWO columns in the CSV training and validation files:
1. column: Input
2. column: Desired output that corresponds to the input

In the case of time series prediction the desired output to an input at time point t is the value at time point t + 1 because that is what the net shall learn , right?

I'v attached the csv file that corresponds to the lesson 'Train.mbl' from the example I posted.
Just replace the data in the input and output column by your data, leave the first line (the names) unchanged. Ensure that the output column always shows the value that appears next (i.e. one row below) in the input column, just like in the sive wave example. Once you have created your CSV files according to these instructions please try to continue and get back to me in case you are stuck again. Please describe exactly where you are stuck and lets focus on one single next step at a time ;-)

Kind regards

PS: CSV files are now allowed to be attached in the forum.
Attachments
Train.csv
CSV example of the sine wave training lesson
(4.17 KiB) Downloaded 656 times
Thomas Jetter

Post Reply