API Call to find OpenMPI IO Backend

[ax3l]

Hi,

@franzpoeschel and I develop a parallel, domain-science centered I/O layer built on parallel HDF5 (based on MPI-I/O) and ADIOS2 called openPMD.

Our users use OpenMPI and MPICH.
For OpenMPI, we sometimes need to work-around issues depending on the version and MPI-I/O backend, e.g., "for OpenMPI <5 with ROMIO" or "for OpenMPI 3 with OMPIO". The defaults for the MPI-I/O backend vary, depending on: cluster admin configuration options at built time, environment variables and mpirun CLI options.

Is there a runtime API to query the currently used MPI-I/O backend in OpenMPI?

So far, we tried looking at the env variable OMPI_MCA_io, but since it is processed by OpenMPI, contains logical operators (like lists of other options and "not") and reacts differently depending on built-time configure options of OpenMPI, it would be safer if we could query the MPI-I/O backend with an OpenMPI API.

attn @edgargabriel :)

[ax3l]

Ping @edgargabriel @ggouaillardet for guidance/help :)
cc @qkoziol because it's HDF5 related :)

[ggouaillardet]

I do not think there is one yet, but I'll try. Keep in mind the backend is not "one for all" but on a per-file basis. For example, ompio is the default backend but in the previous versions, ROMIO was used for files on a Lustre filesystem.

[edgargabriel]

@axl3 I was actually looking in to this a few days, I don't think there is a clean and easy solution, the information is not exposed to the application layer. In addition, the method would have to work out of the box with older versions of Open MPI, otherwise it is worthless (i.e. adding now a new function that would provide the information doesn't resolve the issue for you).

I tried de-referencing the MPI_File pointer on the application layer to determine selected component , but the compiler doesn't let me do that (error: dereferencing pointer to incomplete type ‘struct ompi_file_t).

[jsquyres]

@ax3l Probably the best we could do is offer new functionality that would set an info key on each MPI_File that indicates which module was selected at run time. That would offer a portable mechanism to retrieve the information (i.e., MPI_File_get_info()), but it would still obviously be an Open MPI-specific info key.

This would help for future versions of Open MPI, but wouldn't help with older versions.

[qkoziol]

Ping @edgargabriel @ggouaillardet for guidance/help :) cc @qkoziol because it's HDF5 related :)

https://soundcloud.com/user-24030555/you-ranglurch

I think the info key idea is the correct route. There is a bug in the HDF5 issue tracker to fix this currently being broken (HDFGroup/hdf5#3025), but once that's working correctly again, returning a (new) info key is straightforward.

Tagging @derobins and @fortnern for visibility also.

[qkoziol]

@axl3 I was actually looking in to this a few days, I don't think there is a clean and easy solution, the information is not exposed to the application layer. In addition, the method would have to work out of the box with older versions of Open MPI, otherwise it is worthless (i.e. adding now a new function that would provide the information doesn't resolve the issue for you).

I tried de-referencing the MPI_File pointer on the application layer to determine selected component , but the compiler doesn't let me do that (error: dereferencing pointer to incomplete type ‘struct ompi_file_t).

Is the info key something you could implement?

[edgargabriel]

I can probably do that, but it might take a week or two until I get to it

[jsquyres]

This raises another obvious question: should we do this kind of thing for other MPI handles? E.g., should we have one or more info keys on communicators that indicate what kind of PML and MTL/BTLs were selected for that communicator?

Or, should we use the MPI_T interface to allow users to get the value of the io MCA param (for example) on a give MPI File handle (and the pml and mtl or btl params on MPI comm handles, ... etc.)?

[qkoziol]

I can probably do that, but it might take a week or two until I get to it

Super, thanks!

[qkoziol]

I don't have a super strong opinion here, but would lean toward the info keys.

Tagging @mahermanns & @wrwilliams for the MPI Tools perspective.

[ggouaillardet]

@ax3l FWIW and meanwhile, here is a hack you might find helpful:

note Open MPI must be built with --with-devel-headers

#define _GNU_SOURCE
#include <dlfcn.h>
#include <mpi.h>
#include <openmpi/ompi/file/file.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    MPI_File fd;
    MPI_File_open(MPI_COMM_WORLD, "/tmp/foo.txt", MPI_MODE_RDONLY, MPI_INFO_NULL, &fd);
    Dl_info info;
    dladdr(((ompi_file_t *)fd)->f_io_selected_module.v2_0_0.io_module_file_open, &info);
    printf("%s\n", info.dli_sname);
    MPI_File_close(&fd);
    MPI_Finalize();
    return 0;
}

dladdr() will return (so to speak) a function name that either contains ompio or romio (I could not test the later though).

[wrwilliams]

I don't have a super strong opinion here, but would lean toward the info keys.

Tagging @mahermanns & @wrwilliams for the MPI Tools perspective.

MPI_T gives two potential advantages over info keys as I see it:

1. Any costs associated with providing these info keys/control variables can be avoided for all handles where a user doesn't explicitly care.
2. Implementing these as control variables means that, if any such backend parameters could be altered, there's a mechanism to allow that.

If neither of these matter, info keys should be fine and will be more straightforward to use; certainly it makes sense to implement first with info keys and see whether the costs are acceptable.