CUDA support for default Open-MPI implementation

[tbrandvik]

Environment:

• AWS ParallelCluster / CfnCluster version aws-parallelcluster-2.4.0
• OS: centos7
• Scheduler: slurm
• Master instance type: c5n.large
• Compute instance type: p3.2xlarge

Bug description and how to reproduce:
The default Open-MPI (/opt/amazon/efa) is not compiled with CUDA support.

Steps to reproduce:

1. Run ompi_info --parsable --all | grep mpi_built_with_cuda_support:value on a compute node with a GPU

The output is as follows, indicating that CUDA support is disabled

mca:mpi:base:param:mpi_built_with_cuda_support:value:false

Is there a particular reason why CUDA support is not enabled or would it be possible to compile the default Open-MPI version to support CUDA in future releases?

[sean-smith]

Thanks for pointing this out, we install openmpi from the efa installers, which in turn build the rpm's. We can add cuda support in a future release, until then you'll need to compile from source:

See https://www.open-mpi.org/faq/?category=buildcuda

[tbrandvik]

Perfect - thanks for the quick response!

[tbrandvik]

I just wanted to check if CUDA support is enabled in the recent 2.4.1 release?

[enrico-usai]

Hi @tbrandvik
you can see the details of each release in the Release Notes.

Unfortunately the CUDA support has not been added yet.

[tbrandvik]

Hi @enrico-usai,

I wanted to check if there are any plans to add this in upcoming releases?

Thanks,
Tobias

[oplatek]

Hi, is there any update?

[Srijan1214]

Hi,
I am kind of new with AWS and was hoping if I could get some help with compiling openmpi with Cuda support from source. As far as I know, I can only build into the head node using AWS batch. How can I build this for all of the compute instances?

[wzamazon]

As far as I know, I can only build into the head node using AWS batch. How can I build this for all of the compute instances?

The compute instance and the head node have shared disk (such as /home). So if you compile open mpi on head node under your home directory, the compute node should have access to that.

[enrico-usai]

Resolving this since as part of 3.9.0 release we included EFA installer 1.30.0 that contains both openmpi40-aws-4.1.6-2 and openmpi50-aws-5.0.0-11.

Open MPI 4 from EFA installer does not support CUDA-awareness. Open MPI 5 does.