RuntimeError: CUDA error: device-side assert triggered

[tahirashehzadi]

tensor([70, 81, 32, 1], device='cuda:0')
Traceback (most recent call last):
File "main50.py", line 565, in
main(args)
File "main50.py", line 418, in main
train_stats = train_one_epoch_burnin(
File "/netscratch/shehzadi/Rego-semi/aH-semi/engine.py", line 377, in train_one_epoch_burnin
loss_dict = train_hybrid(
File "/netscratch/shehzadi/Rego-semi/aH-semi/engine.py", line 54, in train_hybrid
loss_dict = criterion(outputs, targets)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/netscratch/shehzadi/Rego-semi/aH-semi/models/deformable_detr.py", line 457, in forward
indices = self.matcher(outputs_without_aux, targets)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/netscratch/shehzadi/Rego-semi/aH-semi/models/matcher.py", line 176, in forward
cost_class = pos_cost_class[:, tgt_ids] - neg_cost_class[:, tgt_ids]
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::CUDAError'
what(): CUDA error: device-side assert triggered

I am getting this error.