Error training efficientDet "RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"

[LordSyd]

Hi,

I am having a problem custom training efficientDet.
This is my Error Output:
`loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loaded pretrained weights for efficientnet-b0
Epoch: 1/100. Iteration: 398/398. Cls loss: 0.31673. Reg loss: 0.31835. Batch loss: 0.63508 Total loss: 1.05050: 100%|███████████████████████████████████████████████████████████████████████| 398/398 [04:18<00:00, 1.54it/s]
Epoch: 1/100. Classification loss: 0.39607. Regression loss: 0.40424. Total loss: 0.80032
/home/fhcampus01/Documents/GitHub/efficientdet/src/model.py:251: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
if len(inputs) == 2:
/home/fhcampus01/Documents/GitHub/efficientdet/src/utils.py:84: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
image_shape = np.array(image_shape)
/home/fhcampus01/Documents/GitHub/efficientdet/src/utils.py:96: TracerWarning: torch.from_numpy results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
anchors = torch.from_numpy(all_anchors.astype(np.float32))
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
File "/home/fhcampus01/Documents/GitHub/efficientdet/train.py", line 189, in
train(opt)
File "/home/fhcampus01/Documents/GitHub/efficientdet/train.py", line 168, in train
torch.onnx.export(model.module, dummy_input,
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 506, in export
_export(
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1548, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1113, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 989, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 893, in _trace_and_get_graph_from_model
trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 1268, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 127, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 118, in wrapper
outs.append(self.inner(*trace_inputs))
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/fhcampus01/Documents/GitHub/efficientdet/src/model.py", line 276, in forward
transformed_anchors = self.clipBoxes(transformed_anchors, img_batch)
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/fhcampus01/Documents/GitHub/efficientdet/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/fhcampus01/Documents/GitHub/efficientdet/src/utils.py", line 60, in forward
boxes[:, :, 2] = torch.clamp(boxes[:, :, 2], max=width)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument max in method wrapper_CUDA_clamp_Tensor)
`
I honestly am at a loss what to do...

Would appreciate any help.

Kind regards,
Daniel

[xwwkk]

I get same one, do you solve it