HowtoMPI
This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.
General forewords
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:
Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI, and the exact same MPI runtime to execute it
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).
Test program
This ring.c test does the classic MPI "print my rank and host names" helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
void ringSend(int me, int proc) {
int i;
int field = -1;
MPI_Status status;
MPI_Barrier(MPI_COMM_WORLD);
int fieldTx;
char name[MPI_MAX_PROCESSOR_NAME];
int namelen;
MPI_Get_processor_name(name, &namelen);
if (me==0) {
field = 0;
fieldTx = field + 1;
MPI_Send(&fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);
MPI_Recv(&field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &status);
printf("Rank %d on %s: field rx = %d, field tx = %d\n", me, name, field, fieldTx);
}
else {
MPI_Recv(&field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &status);
fieldTx = field + 1;
MPI_Send(&fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);
printf("Rank %d on %s: field rx = %d, field tx = %d\n", me, name, field, fieldTx);
}
}
int main(int argc, char **argv) {
int proc, me;
MPI_Init (&argc, & argv);
MPI_Comm_size (MPI_COMM_WORLD, &proc);
MPI_Comm_rank (MPI_COMM_WORLD, &me);
ringSend(me, proc);
MPI_Finalize ();
}
An example output, excepting the usual complaints from openmpi, might look like
erik-k@cyclops ~/mpitester $ cat hostlist cyclops gorgon erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring -------------------------------------------------------------------------- WARNING: No preset parameters ... ... (blah blah blah) ... -------------------------------------------------------------------------- 1 on gorgon.stor: field rx = 1, field tx = 2 3 on gorgon.stor: field rx = 3, field tx = 4 5 on gorgon.stor: field rx = 5, field tx = 6 0 on cyclops.stor: field rx = 6, field tx = 1 2 on cyclops.stor: field rx = 2, field tx = 3 4 on cyclops.stor: field rx = 4, field tx = 5
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.
The --map-by node is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.
The use of --prefix is key as, without using a cluster scheduler, there is no other way to invoke the correct orted and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,
--prefix $(which orted | sed -e 's/\/bin\/orted//') can be used to do this inline.
MPI communication
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using
-mca btl tcp_if_include=x
or
-mca btl tcp_if_exclude=x
arguments.
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.
Tested environments/combinations
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed & the actual output from mpirun is not included.
Running on the 8-node AXIS cluster
OACISS has a small test cluster designated AXIS run by SLURM. This cluster is visible from the orthus login node via the standard slurm commands (sinfo/squeue/srun/salloc/etc).
By default, when an environment is setup inside a Slurm script, Slurm copies that environment for the invoked processes and this is helpful. However because of differences in the installation of Slurm on the Axis nodes and on Orthus, this causes 'srun -N [2 or more] ./foo.sh' to fail when invoked from Orthus. Thus, successful of use of srun on the axis cluster must be performed from one of the axis nodes.
The nodes are axis1 through axis8. They live only on the private network and share a 10 gigabit interconnect.
Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022)
erik-k@cyclops ~/mpitester $ module list Currently Loaded Modules: 1) gcc/10.1 2) openmpi/4.0.1-gcc10.1 erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c erik-k@cyclops ~/mpitester $ cat hostlist cyclops gorgon erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring (...) erik-k@cyclops ~/mpitester $ echo $? 0
Power9 / Openmpi-4.0.1-llvm8.0.1
erik-k@cyclops ~/mpitester $ module list Currently Loaded Modules: 1) llvm/8.0.1 2) openmpi/4.0.1-llvm8.0.1 erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring (...) erik-k@cyclops ~/mpitester $ echo $? 0