Importing CSV file doesn't work properly

Submitted by Oedsen on Sat, 06/05/2010 - 08:13

Using Encog in Java

Hi,

I am trying to import my training data in the Encog workbench, but I can't seem to get it right. I try to import all double values, but the training set contains only integers. Also, every row in the CSV file gets imported twice in Encog. Is there any way to solve this problem? I don't want to manually correct all the input, since it is a lot of data.

Regards,
Oedsen

Submitted by jeffheaton on Sun, 06/06/2010 - 16:56.

I tried it and it worked just fine. I did a CSV with:

1.0,0.1,0.001
2.0,0.2,0.002
3.0,0.3,0.003
4.0,0.4,0.004

It imported okay.

Submitted by mapomapo on Fri, 09/17/2010 - 04:49.

Hi,
I am trying to import a training set from a csv file and I'm having the "double to integer" problem, even using your proposed csv
1.0,0.1,0.001
2.0,0.2,0.002
3.0,0.3,0.003
4.0,0.4,0.004

encog workbench imports
1 0 0
2 0 0
3 0 0
4 0 0

I've also tried to fill manually the input table and export the csv. It exports fine, but when I try to import the just-exported one it gives me the same "double to integer" issue.
Any suggestions?

Thanks in advance,
Vito Renò

Submitted by SeemaSingh on Fri, 09/17/2010 - 08:29.

I figured it out. Just checked in a fix to the 2.4.x codebase. The problem was that the workbench was detecting your local decimal point char, which was probably a decimal comma. Jeff did not see it because he is probably running with standard USA settings. I made it so it always assumes decimal point. It would not have worked with a decimal comma file either, as the workbench would have assumed a comma separator as well!

There are now a number of good fixes in 2.4.x. Should we release a 2.4.3 before the 2.5 beta?

Submitted by mapomapo on Fri, 09/17/2010 - 12:23.

Thank you very much for your answer!

Vito

P.S.
I tried to change my decimal settings (I'm using mac osx 10.6.4) but it wasn't working. I changed the whole OS language and then it worked fine. This solution is not the best for me, so I spent some time reading the code and "surfing" into the sources...I resolved changing line 84 in org.encog.util.csv.CSVFormat.java putting a return '.'; instead of return ch;
Now it just works fine.

Submitted by jeffheaton on Sat, 09/18/2010 - 07:17.

I switched one of my computer's to have country settings for various countries that represent decimal comma, rather than point. And I have the same problem. Seema's change seemed to correct it for me. I will look at the line you mentioned above and Seema's change, and we will incorporate this into 2.4.3, which is going to be released next week.

Thanks for the information!

This has come up before. I think I am going to leave one of the development machines in this mode just for testing purposes.

By the way, the new Encog 2.5's workbench has much better support for CSV. You can import/export to either format very easily.