RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-9xcrj8au/aten/src/THC/THCBlas.cu:259

[HardikJain02]

I read somewhere it might be data loading issue or it can be pytorch version issue. Can you help to rectify?

Begin to resume from the last checkpoint.
Loading checkpoint from out/total3d/20110611514267/model_best.pth.
Warning: Could not find epoch in checkpoint!
Warning: Could not find min_loss in checkpoint!
Warning: Could not find step in checkpoint!
set() subnet missed.
Checkpoint out/total3d/20110611514267/model_best.pth resumed.
----------------------------------------------------------------------------------------------------
Loading data.
Traceback (most recent call last):
  File "main.py", line 42, in <module>
    demo.run(cfg)
  File "/workspace/hardik/Implicit3DUnderstanding/demo.py", line 147, in run
    est_data = net(data)
  File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/hardik/Implicit3DUnderstanding/models/total3d/modules/network.py", line 97, in forward
    lo_centroid_result, lo_coeffs_result, a_features = self.layout_estimation(data['image'])
  File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/hardik/Implicit3DUnderstanding/models/total3d/modules/layout_estimation.py", line 63, in forward
    cam = self.fc_1(x)
  File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/workspace/hardik/Implicit3DUnderstanding/workspace/.conda/envs/Im3D/lib/python3.6/site-packages/torch/nn/functional.py", line 1406, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: cublas runtime error : the GPU program failed to execute at /tmp/pip-req-build-9xcrj8au/aten/src/THC/THCBlas.cu:259

[Jiaqi-Chen-00]

Hi, did you fix this bug?