RuntimeError: NVTX functions not installed. Are you sure you have a CUDA build?

[zarbuk]

Hello everyone, I have used RF diffusion once, however, once I installed Protein MPNN and AF2 for further processing, I keep getting the error "RuntimeError: NVTX functions not installed. Are you sure you have a CUDA build?" when I try to use RF diffusion, is there any suggestion as to how I can overcome it?

[roccomoretti]

Generally this means that you have the CPU-only build of PyTorch installed in your Conda environment.

First off, do you have an NVIDIA CPU with CUDA support? Usually you can run nvidia-smi to get a report, and see the maximum CUDA version your hardware supports. For the provided environment files, this should be 11.1 or greater. If your max supported CUDA version is less than that, or you don't have an CUDA-capable NVIDIA GPU, then you'll have to adjust your conda installation for it.

If you do have at least CUDA 11.1 support, then try a conda list | grep torch with the RFDiffusion environment activated. You should see "pytorch" as one of the lines. You should be able to also see the detailed version information on that line. If you don't see "cuda" in the version information, then you have the CPU version, and you'll need to install a version of pytorch which has GPU support. You may need to re-install pytorch, trying to get the proper cuda verison. You may want to try the one from the pytorch channel, as that sometimes installs better than the one from the conda-forge channel.

[zarbuk]

Hi, thank you so much. I had Nvidia CPU with cuda support but I had multiple cuda toolkit versions which led to this error. I had cuda toolkit version clashes, just changed the conda env it worked for me, however, I do not know how to go about it for DLbinder design, in all likelihood I will run into this issue again.
Can you suggest how do I go about it ?

[roccomoretti]

Once you have your Conda environment set up, you should be able to keep using it as-is.

If you need to re-install the Conda environment (e.g. for a different user or on a different machine), you'll unfortunately need to debug the CUDA version issues again. But if you keep track of which re-installation commands you used for a particular machine/CUDA version, that hopefully should be easier next time.

[zarbuk]

Hello, sorry to bother you , my environment was working fine until it wasn't, I am not sure why .. I ran RFdiffusion to generate adjacency pytorch and secondary structure files and once I got them I started my diffusion protocol and I keep getting this error

I have not changed a thing :

File "/home/zbukhari/RFdiffusion/scripts/../scripts/run_inference.py", line 94, in main
px0, x_t, seq_t, plddt = sampler.sample_step(
File "/home/zbukhari/RFdiffusion/rfdiffusion/inference/model_runners.py", line 664, in sample_step
msa_prev, pair_prev, px0, state_prev, alpha, logits, plddt = self.model(msa_masked,
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zbukhari/RFdiffusion/rfdiffusion/RoseTTAFoldModel.py", line 102, in forward
msa, pair, R, T, alpha_s, state = self.simulator(seq, msa_latent, msa_full, pair, xyz[:,:,:3],
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zbukhari/RFdiffusion/rfdiffusion/Track_module.py", line 420, in forward
msa_full, pair, R_in, T_in, state, alpha = self.extra_block[i_m](msa_full,
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zbukhari/RFdiffusion/rfdiffusion/Track_module.py", line 332, in forward
R, T, state, alpha = self.str2str(msa, pair, R_in, T_in, xyz, state, idx, motif_mask=motif_mask, top_k=0)
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/cuda/amp/autocast_mode.py", line 141, in decorate_autocast
return func(args, **kwargs)
File "/home/zbukhari/RFdiffusion/rfdiffusion/Track_module.py", line 266, in forward
shift = self.se3(G, node.reshape(BL, -1, 1), l1_feats, edge_feats)
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zbukhari/RFdiffusion/rfdiffusion/SE3_network.py", line 83, in forward
return self.se3(G, node_features, edge_features)
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/se3_transformer-1.0.0-py3.9.egg/se3_transformer/model/transformer.py", line 140, in forward
basis = basis or get_basis(graph.edata['rel_pos'], max_degree=self.max_degree, compute_gradients=False,
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/se3_transformer-1.0.0-py3.9.egg/se3_transformer/model/basis.py", line 166, in get_basis
with nvtx_range('spherical harmonics'):
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/contextlib.py", line 119, in enter
return next(self.gen)
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/cuda/nvtx.py", line 59, in range
range_push(msg.format(*args, **kwargs))
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/cuda/nvtx.py", line 28, in range_push
return _nvtx.rangePushA(msg)
File "/home/zbukhari/miniconda3/envs/SE3nv/lib/python3.9/site-packages/torch/cuda/nvtx.py", line 9, in _fail
raise RuntimeError("NVTX functions not installed. Are you sure you have a CUDA build?")
RuntimeError: NVTX functions not installed. Are you sure you have a CUDA build?

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

when I do conda list | grep torch I have these:

conda list | grep torch
pytorch 1.9.1 cpu_py39hc5866cc_3 conda-forge
torchaudio 0.9.1 py39 pytorch
torchvision 0.15.2 cpu_py39h83e0c9b_0

[roccomoretti]

pytorch 1.9.1 cpu_py39hc5866cc_3 conda-forge

The "cpu" there is your issue. It's looking for the CUDA version of pytorch and you don't have it. I'm not sure why it worked for you previously, but it may be that you're using slightly different settings this time (or doing a more extensive computation). I would indeed recommend trying to install the CUDA versions of pytorch if you can. Not only will that speed the computation, RFDiffusion uses NVIDIA's version of se3_transformer, which limits the incentive to make working on CPUs easy. -- That said, I think there's ways of forcing a CPU-only installation, but I'm not personally familiar with them, and I'm not sure how well those work if you also have a CUDA-capable GPU installed.

As mentioned, you likely need to try re-installing pytorch in the conda environment, trying to get the cuda version. You may have luck using -c pytorch on the installation command line to try to force use of the pytorch channel (versus the conda-forge channel it looks like you're currently using).

[zarbuk]

It worked, thanks

[zarbuk]

Thanks for the tip it worked! I just did conda install -c pytorch pytorch

…

On Mon, Apr 8, 2024 at 6:16 AM Rocco Moretti ***@***.***> wrote: pytorch 1.9.1 cpu_py39hc5866cc_3 conda-forge The "cpu" there is your issue. It's looking for the CUDA version of pytorch and you don't have it. I'm not sure why it worked for you previously, but it may be that you're using slightly different settings this time (or doing a more extensive computation). I would indeed recommend trying to install the CUDA versions of pytorch if you can. Not only will that speed the computation, RFDiffusion uses NVIDIA's version of se3_transformer, which limits the incentive to make working on CPUs easy. -- That said, I think there's ways of forcing a CPU-only installation, but I'm not personally familiar with them, and I'm not sure how well those work if you also have a CUDA-capable GPU installed. As mentioned, you likely need to try re-installing pytorch in the conda environment, trying to get the cuda version. You may have luck using -c pytorch on the installation command line to try to force use of the pytorch channel (versus the conda-forge channel it looks like you're currently using). — Reply to this email directly, view it on GitHub <#215 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7D4E65PCGOIYVTPJ2NREB3Y4KKBFAVCNFSM6AAAAABFP32YLWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TANBWGA3TK> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[heinzdoof]

Hey!

I tried the solutions mentioned in the above section. But I am still getting the same error.

conda list | grep torch
gives me the following:
libtorch 2.3.1 cpu_mkl_h0bb0d08_100 conda-forge
pytorch 2.3.1 cpu_mkl_py39h85c4de8_100 conda-forge
pytorch-cuda 12.4 hc786d27_6 pytorch
pytorch-mutex 1.0 cuda pytorch

I have also tried updating all the dependencies in the conda environment. Yet the issue persists. Anybody else experiencing this problem?

[roccomoretti]

It looks like you've installed the CPU versions of the pytorch and libtorch libraries from the conda-forge channel, but you have the pytorch-cuda package from the pytorch channel. This channel difference may be why you're not getting the correct version of pytorch (the pytorch-cuda package is a bit of a meta/signal package which signals that the GPU versions of the pytorch package should be installed.)

I think if you do something like conda install -c pytorch libtorch pytorch, the addition of the -c pytorch channel specification might be enough to re-install pytorch and libtorch from the pytorch conda channel, which should (hopefully) pick up you have the pytorch-cuda package installed and result in you installing the cuda version of pytorch.

If that doesn't work, try uninstalling pytorch & libtorch first, then redoing the install with the -c pytorch option.