HowtoMPI - Revision history

Nic-wiki: /* Test program */

2022-02-03T09:40:16Z

Test program

← Older revision		Revision as of 09:40, 3 February 2022
Line 81:		Line 81:
	The <code>--map-by node</code> is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.		The <code>--map-by node</code> is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.

	The use of <code>--prefix</code> is key as, without using a cluster scheduler, there is no other way to invoke the correct <~~pre~~>orted</~~pre~~> and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,		The use of <code>--prefix</code> is key as, without using a cluster scheduler, there is no other way to invoke the correct <code>orted</code> and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,
	<code> --prefix $(which orted \| sed -e 's/\/bin\/orted//')</code> can be used to do this inline.		<code> --prefix $(which orted \| sed -e 's/\/bin\/orted//')</code> can be used to do this inline.

Nic-wiki: /* General forewords */

2022-02-03T09:34:53Z

General forewords

← Older revision		Revision as of 09:34, 3 February 2022
Line 5:		Line 5:
	There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:		There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:

	<pre>Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI</pre>		<pre>Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI, and the exact same MPI runtime to execute it</pre>

	This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.		This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.

Nic-wiki at 09:34, 3 February 2022

2022-02-03T09:34:19Z

← Older revision		Revision as of 09:34, 3 February 2022
Line 79:		Line 79:
	As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.		As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.

	The <code>--map-by node</code is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.		The <code>--map-by node</code> is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.

	The use of <code>--prefix</code> is key as, without using a cluster scheduler, there is no other way to invoke the correct <pre>orted</pre> and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,		The use of <code>--prefix</code> is key as, without using a cluster scheduler, there is no other way to invoke the correct <pre>orted</pre> and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,
Line 95:		Line 95:

	Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed & the actual output from mpirun is not included.		Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed & the actual output from mpirun is not included.

			=== Running on the 8-node AXIS cluster ===

			OACISS has a small test cluster designated AXIS run by SLURM. This cluster is visible from the orthus login node via the standard slurm commands (sinfo/squeue/srun/salloc/etc).

			By default, when an environment is setup inside a Slurm script, Slurm copies that environment for the invoked processes and this is helpful. However because of differences in the installation of Slurm on the Axis nodes and on Orthus, this causes 'srun -N [2 or more] ./foo.sh' to fail when invoked from Orthus. Thus, successful of use of srun on the axis cluster must be performed from one of the axis nodes.

			The nodes are axis1 through axis8. They live only on the private network and share a 10 gigabit interconnect.

	=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===		=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===

Nic-wiki: /* Test program */

2022-02-03T09:27:23Z

Test program

← Older revision		Revision as of 09:27, 3 February 2022
Line 79:		Line 79:
	As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.		As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.

			The <code>--map-by node</code is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.

	The use of --prefix is key as, without using a cluster scheduler, there is no other way to invoke the correct <pre>orted</pre> and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,		The use of <code>--prefix</code> is key as, without using a cluster scheduler, there is no other way to invoke the correct <pre>orted</pre> and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,
	<code> --prefix $(which orted \| sed -e 's/\/bin\/orted//')</code> can be used to do this inline.		<code> --prefix $(which orted \| sed -e 's/\/bin\/orted//')</code> can be used to do this inline.

Nic-wiki: /* Test program */

2022-02-03T09:19:03Z

Test program

← Older revision		Revision as of 09:19, 3 February 2022
Line 79:		Line 79:
	As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.		As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.


			The use of --prefix is key as, without using a cluster scheduler, there is no other way to invoke the correct <pre>orted</pre> and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,
			<code> --prefix $(which orted \| sed -e 's/\/bin\/orted//')</code> can be used to do this inline.

	== MPI communication ==		== MPI communication ==

Nic-wiki: /* Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) */

2022-02-03T02:05:36Z

Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022)

← Older revision		Revision as of 02:05, 3 February 2022
Line 101:		Line 101:
	cyclops		cyclops
	gorgon		gorgon
	~~erik-k@cyclops ~/mpitester $ vi hostlist~~
	erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring		erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring
	(...)		(...)

Nic-wiki at 02:05, 3 February 2022

2022-02-03T02:05:15Z

@@ Line 102: / Line 102: @@
 gorgon
 erik-k@cyclops ~/mpitester $ vi hostlist
-erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring</pre>
+erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring

Nic-wiki: Created page with "This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware. == General forewords == There are a few key factors that hav..."

2022-02-03T01:58:40Z

Created page with "This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware. == General forewords == There are a few key factors that hav..."

New page

This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.

== General forewords ==

There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:

<pre>Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI</pre>

This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.

Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).

== Test program ==

This ring.c test does the classic MPI "print my rank and host names" helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),

<pre>#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

void ringSend(int me, int proc) {
int i;
int field = -1;
MPI_Status status;

MPI_Barrier(MPI_COMM_WORLD);

int fieldTx;

char name[MPI_MAX_PROCESSOR_NAME];
int namelen;
MPI_Get_processor_name(name, &namelen);

if (me==0) {
field = 0;
fieldTx = field + 1;
MPI_Send(&fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);
MPI_Recv(&field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &status);
printf("Rank %d on %s: field rx = %d, field tx = %d\n", me, name, field, fieldTx);
}
else {
MPI_Recv(&field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &status);
fieldTx = field + 1;
MPI_Send(&fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);
printf("Rank %d on %s: field rx = %d, field tx = %d\n", me, name, field, fieldTx);
}

}

int main(int argc, char **argv) {
int proc, me;

MPI_Init (&argc, & argv);
MPI_Comm_size (MPI_COMM_WORLD, &proc);
MPI_Comm_rank (MPI_COMM_WORLD, &me);

ringSend(me, proc);

MPI_Finalize ();
}</pre>

An example output, excepting the usual complaints from openmpi, might look like

<pre>erik-k@cyclops ~/mpitester $ cat hostlist
cyclops
gorgon
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring
--------------------------------------------------------------------------
WARNING: No preset parameters ...
... (blah blah blah) ...
--------------------------------------------------------------------------
1 on gorgon.stor: field rx = 1, field tx = 2
3 on gorgon.stor: field rx = 3, field tx = 4
5 on gorgon.stor: field rx = 5, field tx = 6
0 on cyclops.stor: field rx = 6, field tx = 1
2 on cyclops.stor: field rx = 2, field tx = 3
4 on cyclops.stor: field rx = 4, field tx = 5</pre>

As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.

== MPI communication ==

For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.

OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using <pre>-mca btl tcp_if_include=x</pre> or <pre>-mca btl tcp_if_exclude=x</pre> arguments.

There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.

== Tested environments/combinations ==

Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed & the actual output from mpirun is not included.

=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===

<pre>erik-k@cyclops ~/mpitester $ module list
Currently Loaded Modules:
1) gcc/10.1 2) openmpi/4.0.1-gcc10.1
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c
erik-k@cyclops ~/mpitester $ cat hostlist
cyclops
gorgon
erik-k@cyclops ~/mpitester $ vi hostlist
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring</pre>