NetworkInfrastructure

OACISS hostnames and speeds

OACISS hardware may have several network connections. If the fully qualified public hostname is 'foo.nic.uoregon.edu' for the host named "foo", then relevant DNS entries are

• foo.nic.uoregon.edu (or foo from within OACISS network) - the public (128.223.202.x) Ethernet interface
• foo.stor - the private (172.17.202.x) Ethernet interface
• foo.ib - the IPoIB interface, if the host has Infiniband

Obviously .stor and .ib are not publicly recognized TLDs. They can only be resolved by OACISS' authoritative DNS server for 128.223.202.0/24, which is only accessible by the same subnet.

The following table describes at a glance the network line rates available host by host. Update 3/25: We are in the process of rolling out 40 and 100 Gbps ethernet to hosts, this table may become out of date.

OACISS Network Speeds
Host	Public Ethernet	Private Ethernet	Infiniband	NFS mount source
Orthus	10G	10G	x	Ethernet
Delphi	10G	100G	x	Ethernet
Instinct	10G	10G	x	Ethernet
Aurora	10G	10G	EDR	IPoIB
Athena	10G	10G	x	Ethernet
Jupiter	10G	100G	EDR	Ethernet
Centaur	10G	10G	x	Ethernet
Minotaur	10G	10G	x	Ethernet
Cyclops	10G	10G	2xEDR	Ethernet
Gorgon	10G	10G	2xEDR	Ethernet
Eagle	10G	10G	2xEDR [64G / pcie]	Ethernet
Illyad	10G	100G	2xEDR	IPoIB
Gilgamesh	10G	100G	2xEDR	IPoIB
Godzilla	10G	40G	EDR	Ethernet
Voltar	10G	10G	EDR	IPoIB
Pegasus	10G	100G	EDR	IPoIB
Vina	10G	10G	x	Ethernet
Pike	10G	10G	x	Ethernet
Grover	1G	1G	x	Ethernet
Omicron	1G	*	x	Ethernet
Sever	10G	*	x	Ethernet
Echs	1G	*	x	Ethernet

• * This system is not on OACISS' rack-local switches

The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.

The internal (172.17 ethernet and the Infiniband) interfaces have no firewall, as the are isolated from the Internet and MPI is... "intolerant" of firewalls.

All public-facing interfaces permit direct login access within the OACISS subnetwork (128.223.202.0/24) but are otherwise a black hole, with exceptions granted to users who request external access on a case-by-case basis.

Machine Room Network Hardware

Ethernet

The NIC Ethernet setup has five switches, four 24-port Dell N4032 10Gbps units and one Dell 32-port z9100 100Gbps unit.

• Dell N4032 x 2: Private network
  • Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)
  • 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)
  • 40GbE to Mnemosyne (Fo1/1/2)
  • 40GbE to OacissBB (Fo2/1/2)
• Dell PowerConnect 8024 + N4032: Public network
  • PC8024 in R34
    • 10Gbps UOnet uplink (Te1/0/1)
    • 2 ports LAG trunk to R35 (Te1/0/21 & 22)

Ethernet interfaces are associated with DNS records:

• The public interface (e.g. FOO.nic.uoregon.edu)
• The private interface with local-only DNS record (e.g. FOO.stor)

EDR Infiniband

OACISS has a 36 port SB7700 EDR Infiniband switch.

Hosts generally have either one or two EDR links to the switch, providing either 100 or 200Gbps connectivity.

Hosts with 200Gbps:

• Cyclops
• Gorgon
• Eagle
• Illyad
• Gilgamesh

Hosts with 100Gbps:

• Aurora (sx-aurora)
• Pegasus
• Godzilla
• Voltar
• Mnemosyne

Hosts within the NIC network will resolve the domain '.ib' in reference to the IPoIB interfaces which provide ~80Gbps connectivity after the encapsulation penalty. Please note, there is no need to specify host.ib in MPI host files order to have MPI utilize the IB interface; MPI will automatically detect and prefer the ultra-high-speed link unless it is explicitly disabled.

Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.

Streisinger Network

Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.