Skip to content

Update OpenMPI documentation #301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
msimberg wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Update OpenMPI documentation #301

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
fdbb441
Update OpenMPI documentation
msimberg Nov 14, 2025
21f78e0
Add link to missing GPU-GPU synchronization issue
msimberg Nov 14, 2025
ece72a4
Remove word
msimberg Nov 14, 2025
941b881
Merge branch 'main' into ompi-docs
bcumming Dec 4, 2025
dea8032
Refactor lnx vs cxi openmpi sections
msimberg Dec 17, 2025
77f8e2b
Add example of broken MPI_Iallreduce with GPU buffers using OpenMPI
msimberg Dec 17, 2025
4f8f353
Merge remote-tracking branch 'origin/main' into ompi-docs
msimberg Dec 17, 2025
20ac9b0
Update prgenv-gnu-openmpi package list
msimberg Dec 19, 2025
210f875
Update note about openmpi containers and intra-node communication
msimberg Dec 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 42 additions & 24 deletions docs/software/communication/openmpi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,11 @@
[Cray MPICH][ref-communication-cray-mpich] is the recommended MPI implementation on Alps.
However, [OpenMPI](https://www.open-mpi.org/) can be used as an alternative in some cases, with limited support from CSCS.

To use OpenMPI on Alps, it must be built against [libfabric][ref-communication-libfabric] with support for the [Slingshot 11 network][ref-alps-hsn].
Support for the [Slingshot 11 network][ref-alps-hsn] is provided by the [libfabric][ref-communication-libfabric] library.

## Using OpenMPI

!!! warning
Building and using OpenMPI on Alps is still [work in progress](https://eth-cscs.github.io/cray-network-stack/).
The instructions found on this page may be inaccurate, but are a good starting point to using OpenMPI on Alps.

!!! todo
Deploy experimental uenv.

!!! todo
Document OpenMPI uenv next to prgenv-gnu, prgenv-nvfortran, and linalg?

OpenMPI is provided through a [uenv][ref-uenv] similar to [`prgenv-gnu`][ref-uenv-prgenv-gnu].
OpenMPI is provided in the [`prgenv-gnu-openmpi`][ref-uenv-prgenv-gnu-openmpi] uenv.
Once the uenv is loaded, compiling and linking with OpenMPI and libfabric is transparent.
At runtime, some additional options must be set to correctly use the Slingshot network.

Expand All @@ -32,27 +22,55 @@ Additionally, the following environment variables should be set:
```bash
export PMIX_MCA_psec="native" # (1)!
export FI_PROVIDER="cxi" # (2)!
export OMPI_MCA_pml="^ucx" # (3)!
export OMPI_MCA_pml="cm" # (3)!
export OMPI_MCA_mtl="ofi" # (4)!
```

1. Ensures PMIx uses the same security domain as Slurm. Otherwise PMIx will print warnings at startup.
2. Use the CXI (Slingshot) provider.
3. Use anything except [UCX](https://openucx.org/documentation/) for [point-to-point communication](https://docs.open-mpi.org/en/v5.0.x/mca.html#selecting-which-open-mpi-components-are-used-at-run-time). The `^` signals that OpenMPI should exclude all listed components.
3. Use CM for [point-to-point communication](https://docs.open-mpi.org/en/v5.0.x/mca.html#selecting-which-open-mpi-components-are-used-at-run-time).
4. Use libfabric for the [Matching Transport Layer](https://docs.open-mpi.org/en/v5.0.x/mca.html#frameworks).

!!! info "CXI provider does all communication through the network interface cards (NICs)"
When using the libfabric CXI provider, all communication goes through NICs, including intra-node communication.
This means that intra-node communication can not make use of shared memory optimizations and the maximum bandwidth will not be severely limited.
This means that intra-node communication can not make use of shared memory optimizations and the maximum bandwidth will be severely limited.
Copy link
Member

@RMeli RMeli Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd move this note higher up (at the beginning of the section), so that the limitation is clear from the start, and add a link to the next section in this note. Something like "For an experimental workaround to this limitation, see 'Using the external LINKx provider'".

Copy link
Collaborator Author

@msimberg msimberg Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the other refactoring comment. I didn't explicitly move this comment further up but added a general intro that compares and explains the CXI and LINKx providers.


### Using the experimental LINKx provider

The default configuration routes all communication through the NICs.
While performance may sometimes be acceptable, this mode does not make full use of the much higher intra-node bandwidth available on Grace-Hopper nodes.
In particular, GPU-GPU communication is significantly faster when using the appropriate intra-node links.

The experimental [LINKx](https://ofiwg.github.io/libfabric/v2.3.1/man/fi_lnx.7.html) libfabric provider allows composing multiple libfabric providers for inter- and intra-node communication.
The CXI provider can be used for inter-node communication while the shared memory (`shm`) provider can be used to take advantage of xpmem for CPU-CPU communication and GDRCopy for GPU-GPU communication.

!!! warning "The LINKx provider is experimental and may contain bugs, in particular for intra-node communication"
Copy link
Member

@RMeli RMeli Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
!!! warning "The LINKx provider is experimental and may contain bugs, in particular for intra-node communication"
!!! danger "The LINKx provider is experimental and may contain bugs, in particular for intra-node communication"

Copy link
Collaborator Author

@msimberg msimberg Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may actually go the opposite direction... The LINKx provider is experimental and it was buggy until very recently. In my latest tests, however, it's actually been very good. I may be too naive in thinking that the LINKx provider is working well for most applications though. My tests are primarily OSU...

In principle I'd like to promote LINKx as the default/primary choice and point users to CXI-only as a fallback in case they encounter issues.

In the user's shoes, how upset would you be if we recommend LINKx as the first option, only for you to find out that your application crashes with it? Or conversely, how upset would you be if you play it safe with CXI as the default choice, and then you figure out later you've been leaving a lot of performance on the table by not using LINKx even though it works just fine?

Copy link
Member

@RMeli RMeli Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point. My suggestion was mainly triggered by " Always validate your results to ensure MPI is working correctly.", which to me also means that it could lead to wrong results (not only crash).

In principle I'd like to promote LINKx as the default/primary choice and point users to CXI-only as a fallback in case they encounter issues.

Maybe we can rephrase the whole admonition to make it clear that this is the approach we suggest to try (for performance reasons, and because it works for our tests), but that it is still experimental and therefore there might be risks involved? As it is phrased now, the whole admonition sounds quite negative (hence my "danger" suggestion).

Copy link
Collaborator Author

@msimberg msimberg Dec 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given e.g. the problems @jpcoles-cscs had in his tests with lnx, I'll keep it as the potentially-better-but-secondary option and make it a danger as you proposed. I'll slightly restructure the intro though to present the two options a bit more equally.

Copy link
Collaborator Author

@msimberg msimberg Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RMeli I pushed a small refactoring in dea8032. Would you mind having another look?


A patch has been included in the [`prgenv-gnu-openmpi`][ref-uenv-prgenv-gnu-openmpi] uenv for [this LINKx issue](https://github.com/ofiwg/libfabric/issues/11231).
However, the patch may be incomplete and other issues may still be present.
Always validate your results to ensure MPI is working correctly.

To use the LINKx provider, set `--mpi=pmix`, as without the LINKx provider.
Additionally, set the following environment variables:

```bash
export PMIX_MCA_psec="native"
export FI_PROVIDER="lnx" # (1)!
export FI_LNX_PROV_LINKS="shm+cxi:cxi0|shm+cxi:cxi1|shm+cxi:cxi2|shm+cxi:cxi3" # (2)!
export FI_SHM_USE_XPMEM=1 # (3)!
export OMPI_MCA_pml="cm"
export OMPI_MCA_mtl="ofi"
export OMPI_MCA_mtl_ofi_av=table # (4)!
```

Libfabric has a new [LINKx](https://ofiwg.github.io/libfabric/v2.1.0/man/fi_lnx.7.html) provider, which allows using different libfabric providers for inter- and intra-node communication.
This provider is not as well tested, but can in theory perform better for intra-node communication, because it can use shared memory.
To use the LINKx provider, set the following, instead of `FI_PROVIDER=cxi`:
1. Use the libfabric LINKx provider, to allow using different libfabric providers for inter- and intra-node communication.
2. Specify which providers LINKx should use.
Use the shared memory provider for intra-node communication and the CXI (Slingshot) provider for inter-node communication.
Choose one of the four available NICs on a node in a round-robin fashion.
3. Explicitly use xpmem for CPU-CPU communication.
The default is to use CMA.
4. The LINKx provider requires this option to be set. TODO: Better explanation?

```bash
export FI_PROVIDER="lnx" # (1)!
export FI_LNX_PROV_LINKS="shm+cxi" # (2)!
```
## Known issues

1. Use the libfabric LINKx provider, to allow using different libfabric providers for inter- and intra-node communication.
2. Use the shared memory provider for intra-node communication and the CXI (Slingshot) provider for inter-node communication.
TODO: async collectives don't work.
4 changes: 4 additions & 0 deletions docs/software/prgenv/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ CSCS provides "programming environments" on Alps vClusters that provide compiler
Provides compilers, MPI, tools and libraries built around the GNU compiler toolchain.
It is the go to programming environment on all systems and target node types, that is it is the first that you should try out when starting to compile an application or create a python virtual environment.

- :fontawesome-solid-layer-group: [__prgenv-gnu-openmpi__][ref-uenv-prgenv-gnu-openmpi] uenv

Same as [`prgenv-gnu`][ref-uenv-prgenv-gnu] but with OpenMPI instead of Cray MPICH.

- :fontawesome-solid-layer-group: [__prgenv-nvfortran__][ref-uenv-prgenv-nvfortran] uenv

Provides a set of tools and libraries for building applications that need the NVIDIA Fortran compiler, commonly required for OpenACC and CUDA-Fortran applications.
Expand Down
18 changes: 18 additions & 0 deletions docs/software/prgenv/prgenv-gnu-openmpi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[](){#ref-uenv-prgenv-gnu-openmpi}
# prgenv-gnu-openmpi

The `prgenv-gnu-openmpi` uenv is otherwise similar to [`prgenv-gnu`][ref-uenv-prgenv-gnu] except it provides OpenMPI instead of Cray MPICH.

!!! warning "OpenMPI is not officially supported on CSCS systems"
Cray MPICH is the preferred, and officially supported, MPI implementation on CSCS systems.
OpenMPI is provided on a best effort basis.
While most applications should work correctly with OpenMPI, there may be missing features, broken functionality, or bad performance compared to Cray MPICH.
Issues are best reported upstream, but CSCS is happy to help facilitate and coordinate issue reporting if you [get in touch][ref-get-in-touch].

Use of the uenv is similar to [`prgenv-gnu`][ref-uenv-prgenv-gnu].
See the [OpenMPI documentation][ref-communication-openmpi] for important information on configuring OpenMPI to take advantage of the Slingshot network.

### Versions

=== "25.11"
TODO
2 changes: 2 additions & 0 deletions docs/software/prgenv/prgenv-gnu.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ It is the go to programming environment on all systems and target node types, th

The [`linalg`][ref-uenv-linalg] environment is similar to prgenv-gnu, with additional linear algebra and mesh partitioning algorithms.

The [`prgenv-gnu-openmpi`][ref-uenv-prgenv-gnu-openmpi] is otherwise similar to `prgenv-gnu`, but provides OpenMPI instead of Cray MPICH.

[](){#ref-uenv-prgenv-gnu-versioning}
## Versioning

Expand Down