NetworkInfrastructure

OACISS hostnames and speeds

OACISS hardware may have several network connections. If the fully qualified public hostname is 'foo.nic.uoregon.edu' for the host named "foo", then relevant DNS entries are

• foo.nic.uoregon.edu - the public (128.223.202.x) Ethernet interface
• foo.stor - the private (172.17.202.x) Ethernet interface
• foo.ib - the IPoIB interface, if the host has Infiniband

Obviously .stor and .ib are not publicly recognized TLDs. They can only be resolved by OACISS' authoritative DNS server for 128.223.202.0/24, which is only accessible by the same subnet.

The following table describes at a glance the network line rates available host by host.

OACISS Network Speeds
Host	Ethernet speed	IB speed	NFS mount source
Orthus	10G	x	Ethernet
Delphi	10G	x	Ethernet
Instinct	10G	x	Ethernet
Aurora	10G	EDR	IPoIB
Centaur	10G	x	Ethernet
Minotaur	10G	x	Ethernet
Cyclops	10G	2xEDR	Ethernet
Gorgon	10G	2xEDR	Ethernet
Eagle	10G	2xEDR [64G / pcie]	Ethernet
Illyad	10G	2xEDR	IPoIB
Gilgamesh	10G	2xEDR	IPoIB
Godzilla	10G	EDR	Ethernet
Voltar	10G	EDR	IPoIB
Pegasus	10G	EDR	IPoIB
Vina	10G	x	Ethernet
Pike	10G	x	Ethernet
Compute: KNL Grover	1G	x	Ethernet

The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.

Network Hardware

Ethernet

The NIC Ethernet setup has four switches, two per cabinet, one each for private/storage and public networks.

• Dell N4032 x 2: Private network
  • Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)
  • 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)
  • 40GbE to Mnemosyne (Fo1/1/2)
• Dell PowerConnect 8024 + N4032: Public network
  • PC8024 in R34
    • 10Gbps UOnet uplink (Te1/0/1)
    • 2 ports LAG trunk to R35 (Te1/0/21 & 22)

Ethernet interfaces are associated with DNS records:

• The public interface (e.g. FOO.nic.uoregon.edu)
• The private interface with local-only DNS record (e.g. FOO.stor)

Infiniband network

OACISS has a 36 port SB7700 EDR Infiniband switch.

Hosts generally have either one or two EDR links to the switch, providing either 100 or 200Gbps connectivity.

Hosts with 200Gbps:

• Cyclops
• Gorgon
• Eagle

Hosts with 100Gbps:

• Aurora (sx-aurora)
• Illyad
• Gilgamesh
• Pegasus
• Godzilla
• Mnemosyne

Hosts within the NIC network will resolve the domain '.ib' in reference to the IPoIB interfaces which provide ~80Gbps connectivity after the encapsulation penalty. Please note, there is no need to specify host.ib in order to have MPI utilize the IB interface; MPI will automatically detect and prefer the ultra-high-speed link unless it is explicitly disabled.

Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.