Floating Point Exception (core dumped)

[ghost]

Hi,

I was able to build the caffe version. Created two directories train and test, in a folder named seaquest, having (0,1,2,3,4) and (0) folders respectively.

On running ../train_lstm.sh 18 0 , I get this error (see screenshot below):

[tknandu]

The same error occurred for me as well. Could you kindly look into it at the earliest.

[junhyukoh]

Hi,
Have you checked if the folder names are in the following format?
./[game name]/train/[%04d]/[%05d].png
An example would be

/seaquest/train/0000/00000.png
/seaquest/train/0000/00001.png
/seaquest/train/0000/...
/seaquest/train/0001/00000.png
/seaquest/train/0001/00001.png
/seaquest/train/0001/...

[tknandu]

Hi,

Naming the folders and files to ensure they are 0-padded ensured that all the episodes & images are loaded. However, after the loading, a Floating Point Exception is still raised.

Is it possible for you to share the sampled trajectories data you have generated (for Seaquest)?

[ghost]

[tknandu]

As shown in the screenshot, even after the episodes and images are successfully loaded, a FloatingPoint Exception is raised. Any idea how to resolve this?

[junhyukoh]

I'm sorry for my late response.
I will look into this issue after ICML deadline (2/5).

[junhyukoh]

This actually works fine with my data.
I just uploaded some example data (2 episodes for train/test).
Can you check if the example works for you?

[ghost]

Hi.. Yes! It works. We had a problem with the GPU memory limit of 2GB on using a GTX 680. The network has 9 million parameters and needs 8+ GB of GPU memory. We ran it on a Titan X and the network trains!
Thanks for the help.

[tknandu]

Hi,

Currently training the LSTM network with 1-step prediction on a TitanX GPU with 12 GB GPU RAM for around 1.5 million iterations on Seaquest. Could you give an estimate of the training time?

Also, could you provide a trained model for 1-step prediction on Seaquest to compare results?