New LSTM error!

[lqniunjunlper]

When lstm run more than 2000 batches in first iteration, the error (float point exception) occurred! some print info as following:

I0909 17:38:59.281976 23954 TrainerInternal.cpp:162] Batch=2000 samples=128000 AvgCost=0.239344 CurrentCost=0.201116 Eval: classification_error_evaluator=0.0999063 CurrentEval: classification_error_evaluator=0.0798125
I0909 17:40:21.527977 23954 Tester.cpp:111] Test samples=25000 cost=0.184603 Eval: classification_error_evaluator=0.07068
.............................................................................................................................. .........................................................................................................................................................................................................................................................................................................................................................................................................................Current Layer forward/backward stack is
LayerName: lstmemory_0
LayerName: fc_layer_0
LayerName: embedding_0
LayerName: word
*** Aborted at 1473414360 (unix time) try "date -d @1473414360" if you are using GNU date ***
Current Layer forward/backward stack is
PC: @ 0x7f0182f0ab25 __ieee754_exp
Current Layer forward/backward stack is
*** SIGFPE (@0x7f0182f0ab25) received by PID 23954 (TID 0x7f017130c700) from PID 18446744071611394853; stack trace: ***
Current Layer forward/backward stack is
@ 0x7f018414d710 (unknown)
Current Layer forward/backward stack is
@ 0x7f0182f0ab25 __ieee754_exp
Current Layer forward/backward stack is
@ 0x7f0182f20b52 __GI___exp
Current Layer forward/backward stack is
@ 0x800235 hppl::tanh()
Current Layer forward/backward stack is
@ 0x587383 paddle::LstmCompute::forwardOneSequence<>()
Current Layer forward/backward stack is
@ 0x58788a paddle::LstmCompute::forwardBatch<>()
Current Layer forward/backward stack is
@ 0x581bdc paddle::LstmLayer::forwardBatch()
Current Layer forward/backward stack is
@ 0x58521a paddle::LstmLayer::forward()
Current Layer forward/backward stack is
@ 0x614b94 paddle::NeuralNetwork::forward()
Current Layer forward/backward stack is
@ 0x61efe6 paddle::TrainerThread::forward()
Current Layer forward/backward stack is
@ 0x621194 paddle::TrainerThread::computeThread()
Current Layer forward/backward stack is
@ 0x7f01832553d2 execute_native_thread_routine
Current Layer forward/backward stack is
@ 0x7f01841459d1 start_thread
Current Layer forward/backward stack is
@ 0x7f0182a3a8fd clone
/data11/dis_ml/deeplearning/paddle/bin/paddle: line 46: 23954 Floating point exception${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}

[emailweixu]

There are several things you can try:

1. decrease learning_rate
2. Try a different optimization algorithm
   These can be changed in settings(). See http://www.paddlepaddle.org/doc/ui/api/trainer_config_helpers/optimizers.html#settings

[lqniunjunlper]

hi @emailweixu , so your meaning is that larger learning_rate or some optimization algorithms will lead to floating point exception? And why is this? Are there some parameter updates too large or zero ? THX~

[F0REacH]

Hi @NIULQfromNJU . I had similar Issue #46 with LSTMs (internal state grow over time), switched to GRUs and have no issues with them so far. Judging by "An Empirical Exploration of Recurrent Network Architectures" paper they are comparable in terms of accuracy. GRU are stable, because the state cannot grow higher than the maximal input

[lqniunjunlper]

@F0REacH thanks man~