NetworkInfrastructure: Difference between revisions

From OACISS Systems Wiki
Jump to navigation Jump to search
Line 27: Line 27:
[[File:Oaciss_switches.png]]
[[File:Oaciss_switches.png]]


== High speed networks ==
== Infiniband network ==


=== Intel OmniPath ===
OACISS has a 36 port SB7700 EDR Infiniband switch.


OACISS has a 100Gbps Intel OmniPath switch and several nodes with OP hardware:
Hosts generally have either one or two EDR links to the switch, providing either 100 or 200Gbps connectivity.
* Pegasus
* Godzilla
* Voltar
* Delphi
 
They are accessed by $HOST-op; These are stored in /etc/hosts on these machines.
 
=== EDR InfiniBand ===


OACISS has a 36 port SB7700 100Gbps EDR Infiniband switch, with access to the following hosts:
Hosts with 200Gbps:
* Cyclops
* Cyclops
* Gorgon
* Gorgon
* Eagle
* Eagle
Hosts with 100Gbps:
* Aurora (sx-aurora)
* Aurora (sx-aurora)
* Illyad
* Gilgamesh
* Pegasus
* Godzilla
* Mnemosyne
* Mnemosyne


All hosts are available using IPoIB as $HOST-ib; These designations are stored in /etc/hosts
Hosts within the NIC network will resolve the domain '.ib' in reference to the IPoIB interfaces which provide ~80Gbps connectivity after the encapsulation penalty. Please note, there is no need to specify host.ib in order to have MPI utilize the IB interface; MPI will automatically detect and prefer the ultra-high-speed link unless it is explicitly disabled.
 
Currently, Gorgon is disconnected from the switch and is instead linked directly to Cyclops on a private point-point 200Gbps link.


The new AMD hosts Illyad and Gilgamesh are equipped with Mellanox Bluefield 100G network cards, and are currently sharing a point-point 100Gbps link. They are known to each other as illyad-ib and gilgamesh-ib.
Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.

Revision as of 08:40, 20 November 2020

OACISS naming / DNS

The majority of OACISS hardware has two network connections: One which is publicly routable, and one that provides access to the storage/NFS network.

All main nodes except Grover currently have 10Gbps public and private ethernet.

Network Hardware

Ethernet

The NIC Ethernet setup has four switches, two per cabinet, one each for private/storage and public networks.

  • Dell N4032 x 2: Private network
    • Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)
    • 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)
    • 40GbE to Mnemosyne (Fo1/1/2)
  • Dell PowerConnect 8024 + N4032: Public network
    • PC8024 in R34
      • 10Gbps UOnet uplink (Te1/0/1)
      • 2 ports LAG trunk to R35 (Te1/0/21 & 22)

Ethernet interfaces are associated with DNS records:

  • The public interface (e.g. FOO.nic.uoregon.edu)
  • The private interface with local-only DNS record (e.g. FOO.stor)
  • If present, the IPMI/ILO/BMC/etc interface with local-only DNS record (e.g. FOO-ipmi.stor)

Oaciss switches.png

Infiniband network

OACISS has a 36 port SB7700 EDR Infiniband switch.

Hosts generally have either one or two EDR links to the switch, providing either 100 or 200Gbps connectivity.

Hosts with 200Gbps:

  • Cyclops
  • Gorgon
  • Eagle

Hosts with 100Gbps:

  • Aurora (sx-aurora)
  • Illyad
  • Gilgamesh
  • Pegasus
  • Godzilla
  • Mnemosyne

Hosts within the NIC network will resolve the domain '.ib' in reference to the IPoIB interfaces which provide ~80Gbps connectivity after the encapsulation penalty. Please note, there is no need to specify host.ib in order to have MPI utilize the IB interface; MPI will automatically detect and prefer the ultra-high-speed link unless it is explicitly disabled.

Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.

Retrieved from "https://systems.nic.uoregon.edu/internal-wiki/index.php?title=NetworkInfrastructure&oldid=3147"