NetworkInfrastructure: Difference between revisions

From OACISS Systems Wiki
Jump to navigation Jump to search
No edit summary
 
(19 intermediate revisions by one other user not shown)
Line 3: Line 3:
OACISS hardware may have several network connections. If the fully qualified public hostname is 'foo.nic.uoregon.edu' for the host named "foo", then relevant DNS entries are
OACISS hardware may have several network connections. If the fully qualified public hostname is 'foo.nic.uoregon.edu' for the host named "foo", then relevant DNS entries are


* foo.nic.uoregon.edu - the public (128.223.202.x) Ethernet interface
* ''foo.nic.uoregon.edu'' (or ''foo'' from within OACISS network) - the public (128.223.202.x) Ethernet interface
* foo.stor - the private (172.17.202.x) Ethernet interface
* ''foo.stor'' - the private (172.17.202.x) Ethernet interface
* foo.ib - the IPoIB interface, if the host has Infiniband
* ''foo.ib'' - the IPoIB interface, if the host has Infiniband


Obviously .stor and .ib are not publicly recognized TLDs. They can only be resolved by OACISS' authoritative DNS server for 128.223.202.0/24, which is only accessible by the same.
Obviously .stor and .ib are not publicly recognized TLDs. They can only be resolved by OACISS' authoritative DNS server for 128.223.202.0/24, which is only accessible by the same subnet.
 
The following table describes at a glance the network line rates available host by host. Update 3/25: We are in the process of rolling out 40 and 100 Gbps ethernet to hosts, this table may become out of date.


{| border="1" cellspacing="0" cellpadding="5"
{| border="1" cellspacing="0" cellpadding="5"
! colspan=4 align=center | OACISS Network Speeds
! colspan=5 align=center | OACISS Network Speeds
|-
|-
! Host !! Ethernet speed !! IB speed !! NFS mount source
! Host !! Public Ethernet !! Private Ethernet !! Infiniband !! NFS mount source
|-
|-
| [[Compute: Orthus | Orthus]] || 10G || x || Ethernet
| [[Compute: Orthus | Orthus]] || 10G || 10G || x || Ethernet
|-
|-
| [[Compute: Delphi | Delphi]] || 10G || x || Ethernet  
| [[Compute: Delphi | Delphi]] || 10G || 100G || x || Ethernet  
|-
|-
| [[Compute: Instinct | Instinct]] || 10G || x || Ethernet
| [[Compute: Instinct | Instinct]] || 10G || 10G || x || Ethernet
|-
|-
| [[Compute: Aurora | Aurora]] || 10G || 100G || Ethernet
| [[Compute: Athena | Athena]] || 10G || 10G || x || Ethernet
|-
|-
| [[Compute: Centaur | Centaur]] || 10G || x || Ethernet
| [[Compute: Jupiter | Jupiter]] || 10G || 100G || EDR || Ethernet
|-
|-
| [[Compute: Minotaur | Minotaur]] || 10G || x || Ethernet
| [[Compute: Centaur | Centaur]] || 10G || 10G || x || Ethernet
|-
|-
| [[Compute: Cyclops | Cyclops]] || 10G || 200G || Ethernet
| [[Compute: Minotaur | Minotaur]] || 10G || 10G || x || Ethernet
|-
|-
| [[Compute: Gorgon | Gorgon]] || 10G || 200G || Ethernet
| [[Compute: Cyclops | Cyclops]] || 10G || 10G || 2xEDR || Ethernet
|-
|-
| [[Compute: Eagle | Eagle]] || 10G || 200G [64G / pcie] || Ethernet
| [[Compute: Gorgon | Gorgon]] || 10G || 10G || 2xEDR || Ethernet
|-
| [[Compute: Eagle | Eagle]] || 10G || 10G || 2xEDR [64G / pcie] || Ethernet
|-  
|-  
| [[Compute: Illyad | Illyad]] || 10G || 200G || IPoIB
| [[Compute: Illyad | Illyad]] || 10G || 100G || 2xEDR || IPoIB
|-
|-
| [[Compute: Gilgamesh | Gilgamesh]] || 10G || 200G || IPoIB
| [[Compute: Gilgamesh | Gilgamesh]] || 10G || 100G || 2xEDR || IPoIB
|-
|-
| [[Compute: Godzilla | Godzilla]] || 10G || 100G || Ethernet
| [[Compute: Godzilla | Godzilla]] || 10G || 40G || EDR || Ethernet
|-
|-
| [[Compute: Voltar | Voltar]] || 10G || 100G || Ethernet
| [[Compute: Voltar | Voltar]] || 10G || 10G || EDR || IPoIB
|-
|-
| [[Compute: Pegasus | Pegasus]] || 10G || 100G || IPoIB
| [[Compute: Pegasus | Pegasus]] || 10G || 100G || EDR || IPoIB
|-  
|-  
| [[Compute: Vina | Vina]] || 10G || x || Ethernet
| [[Compute: Vina | Vina]] || 10G || 10G || x || Ethernet
|-
|-
| [[Compute: Pike | Pike]] || 10G || x || Ethernet
| [[Compute: Pike | Pike]] || 10G || 10G || x || Ethernet
|-
|-
| [[Compute: KNL Grover]] || 1G || x || Ethernet
| [[Compute: KNL Grover | Grover]] || 1G || 1G || x || Ethernet
|-
| [[Compute: Omicron | Omicron]] || 1G || * || x || Ethernet
|-
| [[Compute: Sever | Sever]] || 10G || * || x || Ethernet
|-
| [[Compute: Echs | Echs]] || 1G || * || x || Ethernet
|}
|}


* * This system is not on OACISS' rack-local switches


The majority of OACISS hardware has two network connections: One which is publicly routable, and one that provides access to the storage/NFS network.
The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.


All main nodes except Grover currently have 10Gbps public and private ethernet.
The internal (172.17 ethernet and the Infiniband) interfaces have no firewall, as the are isolated from the Internet and MPI is... "intolerant" of firewalls.


= Network Hardware =
All public-facing interfaces permit direct login access within the OACISS subnetwork (128.223.202.0/24) but are otherwise a black hole, with exceptions granted to users who request external access on a case-by-case basis.
 
= Machine Room Network Hardware =
 
[[File:Oaciss_switches.png]]


== Ethernet ==
== Ethernet ==


The NIC Ethernet setup has four switches, two per cabinet, one each for private/storage and public networks.
The NIC Ethernet setup has five switches, four 24-port Dell N4032 10Gbps units and one Dell 32-port z9100 100Gbps unit.


* Dell N4032 x 2: Private network
* Dell N4032 x 2: Private network  
** Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)
** Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)
** 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)
** 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)
** 40GbE to Mnemosyne (Fo1/1/2)
** 40GbE to Mnemosyne (Fo1/1/2)
** 40GbE to OacissBB (Fo2/1/2)
* Dell PowerConnect 8024 + N4032: Public network
* Dell PowerConnect 8024 + N4032: Public network
** PC8024 in R34
** PC8024 in R34
Line 72: Line 88:
* The public interface (e.g. ''FOO.nic.uoregon.edu'')
* The public interface (e.g. ''FOO.nic.uoregon.edu'')
* The private interface with local-only DNS record (e.g. ''FOO.stor'')
* The private interface with local-only DNS record (e.g. ''FOO.stor'')
* If present, the IPMI/ILO/BMC/etc interface with local-only DNS record (e.g. ''FOO-ipmi.stor'')


[[File:Oaciss_switches.png]]
== EDR Infiniband ==
 
== Infiniband network ==


OACISS has a 36 port SB7700 EDR Infiniband switch.
OACISS has a 36 port SB7700 EDR Infiniband switch.
Line 86: Line 99:
* Gorgon
* Gorgon
* Eagle
* Eagle
* Illyad
* Gilgamesh


Hosts with 100Gbps:
Hosts with 100Gbps:
* Aurora (sx-aurora)
* Aurora (sx-aurora)
* Illyad
* Gilgamesh
* Pegasus
* Pegasus
* Godzilla
* Godzilla
* Voltar
* Mnemosyne
* Mnemosyne


Hosts within the NIC network will resolve the domain '.ib' in reference to the IPoIB interfaces which provide ~80Gbps connectivity after the encapsulation penalty. Please note, there is no need to specify host.ib in order to have MPI utilize the IB interface; MPI will automatically detect and prefer the ultra-high-speed link unless it is explicitly disabled.
Hosts within the NIC network will resolve the domain '.ib' in reference to the IPoIB interfaces which provide ~80Gbps connectivity after the encapsulation penalty. Please note, ''there is no need to specify host.ib in MPI host files order to have MPI utilize the IB interface; MPI will automatically detect and prefer the ultra-high-speed link unless it is explicitly disabled''.


Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.
Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.
= Streisinger Network =
Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.
[[File:Streisinger_switches.png]]

Latest revision as of 22:42, 30 June 2023

OACISS hostnames and speeds

OACISS hardware may have several network connections. If the fully qualified public hostname is 'foo.nic.uoregon.edu' for the host named "foo", then relevant DNS entries are

  • foo.nic.uoregon.edu (or foo from within OACISS network) - the public (128.223.202.x) Ethernet interface
  • foo.stor - the private (172.17.202.x) Ethernet interface
  • foo.ib - the IPoIB interface, if the host has Infiniband

Obviously .stor and .ib are not publicly recognized TLDs. They can only be resolved by OACISS' authoritative DNS server for 128.223.202.0/24, which is only accessible by the same subnet.

The following table describes at a glance the network line rates available host by host. Update 3/25: We are in the process of rolling out 40 and 100 Gbps ethernet to hosts, this table may become out of date.

OACISS Network Speeds
Host Public Ethernet Private Ethernet Infiniband NFS mount source
Orthus 10G 10G x Ethernet
Delphi 10G 100G x Ethernet
Instinct 10G 10G x Ethernet
Athena 10G 10G x Ethernet
Jupiter 10G 100G EDR Ethernet
Centaur 10G 10G x Ethernet
Minotaur 10G 10G x Ethernet
Cyclops 10G 10G 2xEDR Ethernet
Gorgon 10G 10G 2xEDR Ethernet
Eagle 10G 10G 2xEDR [64G / pcie] Ethernet
Illyad 10G 100G 2xEDR IPoIB
Gilgamesh 10G 100G 2xEDR IPoIB
Godzilla 10G 40G EDR Ethernet
Voltar 10G 10G EDR IPoIB
Pegasus 10G 100G EDR IPoIB
Vina 10G 10G x Ethernet
Pike 10G 10G x Ethernet
Grover 1G 1G x Ethernet
Omicron 1G * x Ethernet
Sever 10G * x Ethernet
Echs 1G * x Ethernet
  • * This system is not on OACISS' rack-local switches

The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.

The internal (172.17 ethernet and the Infiniband) interfaces have no firewall, as the are isolated from the Internet and MPI is... "intolerant" of firewalls.

All public-facing interfaces permit direct login access within the OACISS subnetwork (128.223.202.0/24) but are otherwise a black hole, with exceptions granted to users who request external access on a case-by-case basis.

Machine Room Network Hardware

Oaciss switches.png

Ethernet

The NIC Ethernet setup has five switches, four 24-port Dell N4032 10Gbps units and one Dell 32-port z9100 100Gbps unit.

  • Dell N4032 x 2: Private network
    • Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)
    • 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)
    • 40GbE to Mnemosyne (Fo1/1/2)
    • 40GbE to OacissBB (Fo2/1/2)
  • Dell PowerConnect 8024 + N4032: Public network
    • PC8024 in R34
      • 10Gbps UOnet uplink (Te1/0/1)
      • 2 ports LAG trunk to R35 (Te1/0/21 & 22)

Ethernet interfaces are associated with DNS records:

  • The public interface (e.g. FOO.nic.uoregon.edu)
  • The private interface with local-only DNS record (e.g. FOO.stor)

EDR Infiniband

OACISS has a 36 port SB7700 EDR Infiniband switch.

Hosts generally have either one or two EDR links to the switch, providing either 100 or 200Gbps connectivity.

Hosts with 200Gbps:

  • Cyclops
  • Gorgon
  • Eagle
  • Illyad
  • Gilgamesh

Hosts with 100Gbps:

  • Aurora (sx-aurora)
  • Pegasus
  • Godzilla
  • Voltar
  • Mnemosyne

Hosts within the NIC network will resolve the domain '.ib' in reference to the IPoIB interfaces which provide ~80Gbps connectivity after the encapsulation penalty. Please note, there is no need to specify host.ib in MPI host files order to have MPI utilize the IB interface; MPI will automatically detect and prefer the ultra-high-speed link unless it is explicitly disabled.

Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.

Streisinger Network

Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.

Streisinger switches.png

Retrieved from "https://systems.nic.uoregon.edu/internal-wiki/index.php?title=NetworkInfrastructure&oldid=3470"