RuntimeError while training stage3

[pantuwong]

I got the following error while training the stage 3. Note that this error isn't found in stage1 or stage2.

Traceback (most recent call last):
  File "main_meshgraph.py", line 416, in <module>
    main(args)
  File "main_meshgraph.py", line 214, in main
    loss.backward()
  File "/home/napan/anaconda3/envs/c2f/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/napan/anaconda3/envs/c2f/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: function RasterizeFunctionBackward returned an incorrect number of gradients (expected 11, got 10)

[chaneyddtt]

Hi @pantuwong, may I know the command you are using for training?

[pantuwong]

Hi @pantuwong, may I know the command you are using for training?

python main_meshgraph.py --output_dir logs/stage3 --nEpochs 200 --lr 1e-5 --local_feat --w_arap 10000 --w_dice 1000 --w_dice_refine 100 --w_pose_limit_prior 5000 --resume logs/stage2/model_best.pth.tar --gpu_ids 0

[chaneyddtt]

Are you running the original code or you making some modifications?

[pantuwong]

Are you running the original code or you making some modifications?

I am using original code, but try to train from scratch. Note that when I use pretrained model, this error is also raised.

[chaneyddtt]

Can i know the environment you are using? since I could not reproduce this error.

[pantuwong]

Can i know the environment you are using? since I could not reproduce this error.

Ubuntu 18.04
CUDA 11.7
NVIDIA Driver version 515.43.04

And conda environment.yml can be extracted from this zip file.
environment.zip