HowtoMPI

This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.

General forewords

There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:

Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI

This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.

Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).

Test program

This ring.c test does the classic MPI "print my rank and host names" helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

void ringSend(int me, int proc) {
  int i;
  int field = -1;
  MPI_Status status;

  MPI_Barrier(MPI_COMM_WORLD);

  int fieldTx;

  char name[MPI_MAX_PROCESSOR_NAME];
  int namelen;
  MPI_Get_processor_name(name, &namelen);

  if (me==0) {
    field = 0;
    fieldTx = field + 1;
    MPI_Send(&fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);
    MPI_Recv(&field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &status);
    printf("Rank %d on %s: field rx = %d, field tx = %d\n", me, name, field, fieldTx);
  }
  else {
    MPI_Recv(&field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &status);
    fieldTx = field + 1;
    MPI_Send(&fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);
    printf("Rank %d on %s: field rx = %d, field tx = %d\n", me, name, field, fieldTx);
  }

}

int main(int argc, char **argv) {
  int proc, me;

  MPI_Init (&argc, & argv);
  MPI_Comm_size (MPI_COMM_WORLD, &proc);
  MPI_Comm_rank (MPI_COMM_WORLD, &me);

  ringSend(me, proc);

  MPI_Finalize ();
}

An example output, excepting the usual complaints from openmpi, might look like

erik-k@cyclops ~/mpitester $ cat hostlist
cyclops
gorgon
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring
--------------------------------------------------------------------------
WARNING: No preset parameters ...
... (blah blah blah) ...
--------------------------------------------------------------------------
1 on gorgon.stor: field rx = 1, field tx = 2
3 on gorgon.stor: field rx = 3, field tx = 4
5 on gorgon.stor: field rx = 5, field tx = 6
0 on cyclops.stor: field rx = 6, field tx = 1
2 on cyclops.stor: field rx = 2, field tx = 3
4 on cyclops.stor: field rx = 4, field tx = 5

As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.

The use of --prefix is key as, without using a cluster scheduler, there is no other way to invoke the correct

orted

and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,

--prefix $(which orted | sed -e 's/\/bin\/orted//') can be used to do this inline.

MPI communication

For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.

OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using

-mca btl tcp_if_include=x

or

-mca btl tcp_if_exclude=x

arguments.

There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.

Tested environments/combinations

Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed & the actual output from mpirun is not included.

Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022)

erik-k@cyclops ~/mpitester $ module list
Currently Loaded Modules:
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c
erik-k@cyclops ~/mpitester $ cat hostlist
cyclops
gorgon
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring
 (...)
erik-k@cyclops ~/mpitester $ echo $?
0

Power9 / Openmpi-4.0.1-llvm8.0.1

erik-k@cyclops ~/mpitester $ module list
Currently Loaded Modules:
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring
 (...)
erik-k@cyclops ~/mpitester $ echo $?
0