build_hash CUDA kernel failure: invalid device function. pytorch 1.9.0

[sbelharbi]

hi,
thanks for the code.
i built the extension as mentioned in the readme: python setup.py build_ext --inplace. it went fine.
i run this example that was also provided in the readme,

import torch as th

from crfrnn.crf import CRF

n_categories = 32

class MyCNN(th.nn.Module):
    def __init__(self):
        super(MyCNN, self).__init__()
        self.relu = th.nn.ReLU()
        self.conv1 = th.nn.Conv2d(3, 64, 3, 1, 1)
        self.conv2 = th.nn.Conv2d(64, 64, 3, 1, 1)
        self.final = th.nn.Conv2d(64, n_categories, 3, 1, 1)
        self.crf = CRF(n_ref=3, n_out=32)

    def forward(self, x):
        input = x
        x = self.relu(self.conv1(x))
        x = self.relu(self.conv2(x))
        x = th.softmax(self.final(x), dim=1)
        x = self.crf(x, input)
        return x

img = th.zeros(1, 3, 384, 512, device="cuda:0")
model = MyCNN()
model.to(device="cuda:0")
model(img)

i got this error:

$ CUDA_LAUNCH_BLOCKING=1 python run_example.py 
build_hash CUDA kernel failure: invalid device function
Segmentation fault (core dumped)

the error happens right when calling the cuda extension:

crfrnn_layer/crfrnn/crf.py

Line 101 in d4223d8

return gaussian_filter(R, V, self.kstd[None])

and it does not say much.

i am having trouble with cuda extensions. same thing here with PermutohedralLattice.
i tried another server with different hardware, and got the same error.

any idea how to fix this? does this has something to do with compute capability?
i really appreciate your help

config:
conda virtual env: conda create -n env_test python=3.7
python 3.7.9
pytorch 1.9.0 installed with conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c nvidia
cv2 4.1.2

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

CUDA Version with nvcc-smi: 11.1
gpu: tesla p100 (compute capa. 6.0 )

nvisia-smi:
NVIDIA-SMI 455.32.00
Driver Version: 455.32.00

please let me know if you need more info to reproduce this error.

thanks

[sbelharbi]

the issue build_hash CUDA kernel failure: invalid device function was solved by fixing the nvcc paths.
however, the code above still raises an error, a new one,

Unsupported ref_dim/val_dim combination (5, 32), generate a new dispatch table using 'make_gfilt_dispatch.py'.

closing because the error is not related.
please feel free to add comments on how to fix the new error.
thanks