Is mpirun absolutely necessary to run on a single GPU ?

[vreville]

Hi,

I used to always compile with MPI + CUDA on a local cluster and then use ./idefix or mpirun -np d ./idefix (with d>=2) depending on the number of GPUs wanted. On my local cluster, using ./idefix is no longer working to launch the job on a single GPU. There are several changes that could be responsible (update of nvcc, kokkos, openmpi). Here is the openmpi error message :

--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    opal_init:startup:internal-failure
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/25.1/hpcx-12/270223-rel-1/comm_libs/12.6/hpcx/hpcx-2.21/ompi/share/openmpi/help-opal-runtime.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    orte_init:startup:internal-failure
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/25.1/hpcx-12/270223-rel-1/comm_libs/12.6/hpcx/hpcx-2.21/ompi/share/openmpi/help-orte-runtime: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:startup:internal-failure
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/25.1/hpcx-12/270223-rel-1/comm_libs/12.6/hpcx/hpcx-2.21/ompi/share/openmpi/help-mpi-runtime.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[hal:499019] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

I know it is not a big deal, but I wonder if it is possible to go back to the way it was.

Thanks,

[glesur]

Nothing has changed since v2.0 regarding how MPI is initialised in Idefix. As far as I know, calling Idefix compiled with MPI without mpirun still works (at least with open MPI 4). There might be some MPI implementations for which this doesn't work, though...

The error message seems to indicate that your MPI implementation is not properly installed. You should check what's happening with your system administrator.

[vreville]

Ok thanks, I essentially wanted to be sure that this was not bad practice or deprecated somehow !