NetworkInfrastructure - Revision history

Ekeever1 at 22:42, 30 June 2023

2023-06-30T22:42:14Z

← Older revision		Revision as of 22:42, 30 June 2023
Line 21:		Line 21:
	\|-		\|-
	\| [[Compute: Instinct \| Instinct]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Instinct \| Instinct]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-
	~~\| [[Compute: Aurora \| Aurora]] \|\| 10G \|\| 10G \|\| EDR \|\| IPoIB~~
	\|-		\|-
	\| [[Compute: Athena \| Athena]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Athena \| Athena]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet

Nic-wiki: /* OACISS hostnames and speeds */

2021-07-07T03:19:40Z

OACISS hostnames and speeds

← Older revision		Revision as of 03:19, 7 July 2021
Line 18:		Line 18:
	\| [[Compute: Orthus \| Orthus]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Orthus \| Orthus]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: Delphi \| Delphi]] \|\| 10G \|\| ~~10G~~ \|\| x \|\| Ethernet		\| [[Compute: Delphi \| Delphi]] \|\| 10G \|\| 100G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: Instinct \| Instinct]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Instinct \| Instinct]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet

Nic-wiki: /* OACISS hostnames and speeds */

2021-03-31T21:13:45Z

OACISS hostnames and speeds

← Older revision		Revision as of 21:13, 31 March 2021
Line 61:		Line 61:
	\|}		\|}

	* This system is not on OACISS' rack-local switches		* * This system is not on OACISS' rack-local switches

	The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.		The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.

Nic-wiki: /* OACISS hostnames and speeds */

2021-03-31T21:13:26Z

OACISS hostnames and speeds

← Older revision		Revision as of 21:13, 31 March 2021
Line 40:		Line 40:
	\| [[Compute: Illyad \| Illyad]] \|\| 10G \|\| 100G \|\| 2xEDR \|\| IPoIB		\| [[Compute: Illyad \| Illyad]] \|\| 10G \|\| 100G \|\| 2xEDR \|\| IPoIB
	\|-		\|-
	\| [[Compute: Gilgamesh \| Gilgamesh]] \|\| 10G \|\| ~~10G~~ \|\| 2xEDR \|\| IPoIB		\| [[Compute: Gilgamesh \| Gilgamesh]] \|\| 10G \|\| 100G \|\| 2xEDR \|\| IPoIB
	\|-		\|-
	\| [[Compute: Godzilla \| Godzilla]] \|\| 10G \|\| 40G \|\| EDR \|\| Ethernet		\| [[Compute: Godzilla \| Godzilla]] \|\| 10G \|\| 40G \|\| EDR \|\| Ethernet

Nic-wiki: /* Ethernet */

2021-03-25T23:01:12Z

Ethernet

← Older revision		Revision as of 23:01, 25 March 2021
Line 75:		Line 75:
	== Ethernet ==		== Ethernet ==

	The NIC Ethernet setup has ~~four~~ switches, ~~two per cabinet,~~ one ~~each for private/storage and public networks~~.		The NIC Ethernet setup has five switches, four 24-port Dell N4032 10Gbps units and one Dell 32-port z9100 100Gbps unit.

	* Dell N4032 x 2: Private network		* Dell N4032 x 2: Private network
	** Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)		** Switches are stacked. R35 unit is master (#1) and R34 is failover/member (#2)
	** 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)		** 40Gbps trunk link between racks (Fo1/1/1 and Fo2/1/1)
	** 40GbE to Mnemosyne (Fo1/1/2)		** 40GbE to Mnemosyne (Fo1/1/2)
			** 40GbE to OacissBB (Fo2/1/2)
	* Dell PowerConnect 8024 + N4032: Public network		* Dell PowerConnect 8024 + N4032: Public network
	** PC8024 in R34		** PC8024 in R34

Nic-wiki: /* OACISS hostnames and speeds */

2021-03-25T22:57:56Z

OACISS hostnames and speeds

← Older revision		Revision as of 22:57, 25 March 2021
Line 3:		Line 3:
	OACISS hardware may have several network connections. If the fully qualified public hostname is 'foo.nic.uoregon.edu' for the host named "foo", then relevant DNS entries are		OACISS hardware may have several network connections. If the fully qualified public hostname is 'foo.nic.uoregon.edu' for the host named "foo", then relevant DNS entries are

	* foo.nic.uoregon.edu - the public (128.223.202.x) Ethernet interface		* ''foo.nic.uoregon.edu'' (or ''foo'' from within OACISS network) - the public (128.223.202.x) Ethernet interface
	* foo.stor - the private (172.17.202.x) Ethernet interface		* ''foo.stor'' - the private (172.17.202.x) Ethernet interface
	* foo.ib - the IPoIB interface, if the host has Infiniband		* ''foo.ib'' - the IPoIB interface, if the host has Infiniband

	Obviously .stor and .ib are not publicly recognized TLDs. They can only be resolved by OACISS' authoritative DNS server for 128.223.202.0/24, which is only accessible by the same subnet.		Obviously .stor and .ib are not publicly recognized TLDs. They can only be resolved by OACISS' authoritative DNS server for 128.223.202.0/24, which is only accessible by the same subnet.

	The following table describes at a glance the network line rates available host by host.		The following table describes at a glance the network line rates available host by host. Update 3/25: We are in the process of rolling out 40 and 100 Gbps ethernet to hosts, this table may become out of date.

	{\| border="1" cellspacing="0" cellpadding="5"		{\| border="1" cellspacing="0" cellpadding="5"
	! colspan=4 align=center \| OACISS Network Speeds		! colspan=5 align=center \| OACISS Network Speeds
	\|-		\|-
	! Host !! Ethernet ~~speed~~ !! ~~IB speed~~ !! NFS mount source		! Host !! Public Ethernet !! Private Ethernet !! Infiniband !! NFS mount source
	\|-		\|-
	\| [[Compute: Orthus \| Orthus]] \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Orthus \| Orthus]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: Delphi \| Delphi]] \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Delphi \| Delphi]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: Instinct \| Instinct]] \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Instinct \| Instinct]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: Aurora \| Aurora]] \|\| 10G \|\| EDR \|\| IPoIB		\| [[Compute: Aurora \| Aurora]] \|\| 10G \|\| 10G \|\| EDR \|\| IPoIB
	\|-		\|-
	\| [[Compute: ~~Centaur~~ \| ~~Centaur~~]] \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Athena \| Athena]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: ~~Minotaur~~ \| ~~Minotaur~~]] \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Jupiter \| Jupiter]] \|\| 10G \|\| 100G \|\| EDR \|\| Ethernet
	\|-		\|-
	\| [[Compute: ~~Cyclops~~ \| ~~Cyclops~~]] \|\| 10G \|\| ~~2xEDR~~ \|\| Ethernet		\| [[Compute: Centaur \| Centaur]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: ~~Gorgon~~ \| ~~Gorgon~~]] \|\| 10G \|\| ~~2xEDR~~ \|\| Ethernet		\| [[Compute: Minotaur \| Minotaur]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: Eagle \| Eagle]] \|\| 10G \|\| 2xEDR [64G / pcie] \|\| Ethernet		\| [[Compute: Cyclops \| Cyclops]] \|\| 10G \|\| 10G \|\| 2xEDR \|\| Ethernet
			\|-
			\| [[Compute: Gorgon \| Gorgon]] \|\| 10G \|\| 10G \|\| 2xEDR \|\| Ethernet
			\|-
			\| [[Compute: Eagle \| Eagle]] \|\| 10G \|\| 10G \|\| 2xEDR [64G / pcie] \|\| Ethernet
	\|-		\|-
	\| [[Compute: Illyad \| Illyad]] \|\| 10G \|\| 2xEDR \|\| IPoIB		\| [[Compute: Illyad \| Illyad]] \|\| 10G \|\| 100G \|\| 2xEDR \|\| IPoIB
	\|-		\|-
	\| [[Compute: Gilgamesh \| Gilgamesh]] \|\| 10G \|\| 2xEDR \|\| IPoIB		\| [[Compute: Gilgamesh \| Gilgamesh]] \|\| 10G \|\| 10G \|\| 2xEDR \|\| IPoIB
	\|-		\|-
	\| [[Compute: Godzilla \| Godzilla]] \|\| 10G \|\| EDR \|\| Ethernet		\| [[Compute: Godzilla \| Godzilla]] \|\| 10G \|\| 40G \|\| EDR \|\| Ethernet
	\|-		\|-
	\| [[Compute: Voltar \| Voltar]] \|\| 10G \|\| EDR \|\| IPoIB		\| [[Compute: Voltar \| Voltar]] \|\| 10G \|\| 10G \|\| EDR \|\| IPoIB
	\|-		\|-
	\| [[Compute: Pegasus \| Pegasus]] \|\| 10G \|\| EDR \|\| IPoIB		\| [[Compute: Pegasus \| Pegasus]] \|\| 10G \|\| 100G \|\| EDR \|\| IPoIB
	\|-		\|-
	\| [[Compute: Vina \| Vina]] \|\| 10G \|\| x \|\| Ethernet		\| [[Compute: Vina \| Vina]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
			\|-
			\| [[Compute: Pike \| Pike]] \|\| 10G \|\| 10G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: ~~Pike~~ \| ~~Pike~~]] \|\| ~~10G~~ \|\| x \|\| Ethernet		\| [[Compute: KNL Grover \| Grover]] \|\| 1G \|\| 1G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: ~~KNL Grover~~]] \|\| 1G \|\| x \|\| Ethernet		\| [[Compute: Omicron \| Omicron]] \|\| 1G \|\| * \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: ~~Omicron~~]] \|\| 1G \|\| x \|\| Ethernet		\| [[Compute: Sever \| Sever]] \|\| 10G \|\| * \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: ~~Sever~~]] \|\| ~~10G~~ \|\| x \|\| Ethernet		\| [[Compute: Echs \| Echs]] \|\| 1G \|\| * \|\| x \|\| Ethernet
	\|}		\|}

			* This system is not on OACISS' rack-local switches

	The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.		The NFS has a 40GbE network adapter to the private network and (if data is in RAM or on NVME scratch) can fill any 4 hosts' 10Gbit pipes. Infiniband hosts reliably achieve 12.5/25GBps in the osu_bw microbenchmark. Single-threaded speeds exceeding 25-30Gbps have been difficult to reach using IPoIB, but parallel iperf3 instances have reached speeds of around 75-80Gb/s.

	The internal (172.17 ethernet and the Infiniband) interfaces have no firewall, as the are ~~physically~~ isolated from the Internet and MPI is "intolerant" of firewalls.		The internal (172.17 ethernet and the Infiniband) interfaces have no firewall, as the are isolated from the Internet and MPI is... "intolerant" of firewalls.

	All public-facing interfaces permit access ~~with~~ the OACISS subnetwork (128.223.202.0/24) but are otherwise a black hole, with exceptions granted to users who request external access on a case-by-case basis.		All public-facing interfaces permit direct login access within the OACISS subnetwork (128.223.202.0/24) but are otherwise a black hole, with exceptions granted to users who request external access on a case-by-case basis.

	= Machine Room Network Hardware =		= Machine Room Network Hardware =

Nic-wiki: /* Streisinger Network */

2020-12-24T00:32:33Z

Streisinger Network

← Older revision		Revision as of 00:32, 24 December 2020
Line 110:		Line 110:
	Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.		Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.

	[[File:~~streisinger_switches~~.png]]		[[File:Streisinger_switches.png]]

Nic-wiki at 00:31, 24 December 2020

2020-12-24T00:31:16Z

← Older revision		Revision as of 00:31, 24 December 2020
Line 61:		Line 61:
	All public-facing interfaces permit access with the OACISS subnetwork (128.223.202.0/24) but are otherwise a black hole, with exceptions granted to users who request external access on a case-by-case basis.		All public-facing interfaces permit access with the OACISS subnetwork (128.223.202.0/24) but are otherwise a black hole, with exceptions granted to users who request external access on a case-by-case basis.

	= Network Hardware =		= Machine Room Network Hardware =

			[[File:Oaciss_switches.png]]

	== Ethernet ==		== Ethernet ==
Line 80:		Line 82:
	* The private interface with local-only DNS record (e.g. ''FOO.stor'')		* The private interface with local-only DNS record (e.g. ''FOO.stor'')

	~~[[File:Oaciss_switches.png]]~~		== EDR Infiniband ==

	== Infiniband ~~network~~ ==

	OACISS has a 36 port SB7700 EDR Infiniband switch.		OACISS has a 36 port SB7700 EDR Infiniband switch.
Line 109:		Line 109:

	Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.		Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.

			[[File:streisinger_switches.png]]

Nic-wiki at 00:28, 24 December 2020

2020-12-24T00:28:20Z

← Older revision		Revision as of 00:28, 24 December 2020
Line 52:		Line 52:
	\| [[Compute: Omicron]] \|\| 1G \|\| x \|\| Ethernet		\| [[Compute: Omicron]] \|\| 1G \|\| x \|\| Ethernet
	\|-		\|-
	\| [[Compute: Sever]] \|\| 1G \|\| x \|\| Ethernet		\| [[Compute: Sever]] \|\| 10G \|\| x \|\| Ethernet
	\|}		\|}

Nic-wiki at 04:23, 23 December 2020

2020-12-23T04:23:35Z

← Older revision		Revision as of 04:23, 23 December 2020
Line 105:		Line 105:

	Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.		Planning is underway to connect all hosts to the Infiniband network at at least FDR speed.

			= Streisinger Network =

			Hosts living in Streisinger (Cerberus, Chymera, NUC and Jetson cluster) have 1/10GbE access to each other, and (apparently) 1G access to the outside world.