train_loss is NAN when use resnet-101

[ElbertFang]

Thanks for you great job at first.
I have success training the scst use restnet-152. But some problems happened when I use resnet-101. I save the features token by resnet-101 in another file and use it by absolute path and the train_loss became NAN all the time.
I print some data to find out the problems.
fc_feats, att_feats, labels, masks = tmp
print(fc_feats)
and I get this.
Variable containing:
inf inf inf ... 2.3545e-02 4.4336e-01 4.5093e-02
inf inf inf ... 2.3545e-02 4.4336e-01 4.5093e-02
inf inf inf ... 2.3545e-02 4.4336e-01 4.5093e-02
... ⋱ ...
9.7345e-01 1.5666e+00 1.1705e-01 ... 0.0000e+00 2.7455e-01 3.6576e-04
9.7345e-01 1.5666e+00 1.1705e-01 ... 0.0000e+00 2.7455e-01 3.6576e-04
9.7345e-01 1.5666e+00 1.1705e-01 ... 0.0000e+00 2.7455e-01 3.6576e-04
[torch.cuda.FloatTensor of size 50x2048 (GPU 0)]
Looks like I did't get the true fc feats. But I change nothing in prepro_labels.py and it works well when token the features of resnet-152.

[ruotianluo]

I have no idea. really weird. close it for now.