<?xml version="1.0"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
	<id>https://systems.nic.uoregon.edu/internal-wiki/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=Nic-wiki</id>
	<title>OACISS Systems Wiki - User contributions [en]</title>
	<link rel="self" type="application/atom+xml" href="https://systems.nic.uoregon.edu/internal-wiki/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=Nic-wiki"/>
	<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Special:Contributions/Nic-wiki"/>
	<updated>2026-08-19T02:13:49Z</updated>
	<subtitle>User contributions</subtitle>
	<generator>MediaWiki 1.39.12</generator>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Service:storage&amp;diff=3493</id>
		<title>Service:storage</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Service:storage&amp;diff=3493"/>
		<updated>2023-11-30T01:34:21Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;[[Category:Service]]&lt;br /&gt;
&lt;br /&gt;
OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.&lt;br /&gt;
&lt;br /&gt;
Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.&lt;br /&gt;
&lt;br /&gt;
== Main storage: /home/users ==&lt;br /&gt;
&lt;br /&gt;
All systems in the CC utilize the main home directory NAS server on [[Infrastructure:mnemosyne | Mnemosyne]]. This is a primary storage engine with 12 15TB sas-3 SSDs in a raidz2 redundant storage array providing 128TB of disk in one volume.&lt;br /&gt;
&lt;br /&gt;
Single-thread IO tests yield ~1.8GBps sequential read and ~10K IOPS 4K random r/w.&lt;br /&gt;
&lt;br /&gt;
== Package tree: /packages ==&lt;br /&gt;
&lt;br /&gt;
Package trees for RH7, RH8 and ppc64le are stored on 15TB SAS-12G SSDs in RAID-1.&lt;br /&gt;
&lt;br /&gt;
As packages are subject to almost exclusively read access, they are mounted synchronously.&lt;br /&gt;
&lt;br /&gt;
== High performance storage: /gpfs/gpfs0 ==&lt;br /&gt;
&lt;br /&gt;
OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.&lt;br /&gt;
&lt;br /&gt;
The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.&lt;br /&gt;
&lt;br /&gt;
== BeeGFS storage ==&lt;br /&gt;
&lt;br /&gt;
The majority of nodes have a high performance BeeGFS filesystem available.&lt;br /&gt;
&lt;br /&gt;
The data storage lives on a dedicated host, Alexandria, with 15 x 15TB NVME drives arranged in a 180TB raidz2 pool.&lt;br /&gt;
&lt;br /&gt;
== Upstairs storage ==&lt;br /&gt;
&lt;br /&gt;
Upstairs storage is a 28TB SSD storage pool living on Sphinx, serving Silicon and the ARM hardware cluster (jetsons, xaviers, orins and OD1K).&lt;br /&gt;
&lt;br /&gt;
This storage, and its package tree, are entirely separate from main storage.&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Service:storage&amp;diff=3492</id>
		<title>Service:storage</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Service:storage&amp;diff=3492"/>
		<updated>2023-11-30T01:19:04Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Package tree: /packages */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;[[Category:Service]]&lt;br /&gt;
&lt;br /&gt;
OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.&lt;br /&gt;
&lt;br /&gt;
Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.&lt;br /&gt;
&lt;br /&gt;
== Main storage: /home/users ==&lt;br /&gt;
&lt;br /&gt;
All systems in the CC utilize the main home directory NAS server on [[Infrastructure:mnemosyne | Mnemosyne]]. This is a primary storage engine with 12 15TB sas-3 SSDs in a raidz2 redundant storage array providing 128TB of disk in one volume.&lt;br /&gt;
&lt;br /&gt;
Single-thread IO tests yield ~1.8GBps sequential read and ~10K IOPS 4K random r/w.&lt;br /&gt;
&lt;br /&gt;
== Package tree: /packages ==&lt;br /&gt;
&lt;br /&gt;
Package trees for RH7, RH8 and ppc64le are stored on 15TB SAS-12G SSDs in RAID-1.&lt;br /&gt;
&lt;br /&gt;
As packages are subject to almost exclusively read access, they are mounted synchronously.&lt;br /&gt;
&lt;br /&gt;
== High performance storage: /gpfs/gpfs0 ==&lt;br /&gt;
&lt;br /&gt;
OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.&lt;br /&gt;
&lt;br /&gt;
The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.&lt;br /&gt;
&lt;br /&gt;
== High performance scratch: /storage/warpspeed ==&lt;br /&gt;
&lt;br /&gt;
Mnemosyne hosts a low-latency 8TB scratch volume made from two 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0.&lt;br /&gt;
&lt;br /&gt;
== BeeGFS storage ==&lt;br /&gt;
&lt;br /&gt;
Experimental deployment of a BeeGFS high-performance NFS is underway.&lt;br /&gt;
&lt;br /&gt;
The backing store is a ZFS pool to which are assigned:&lt;br /&gt;
* 2 mirrored 4T NVME drives for ZFS metadata&lt;br /&gt;
* 2 4T NVMEs assigned as cache&lt;br /&gt;
* 12 x 18T disks forming a 150TB volume in RaidZ2&lt;br /&gt;
&lt;br /&gt;
BeeGFS itself has been temporarily assigned a directory on the warpspeed filesystem (nvme mdadm raid1) for its own metadata.&lt;br /&gt;
&lt;br /&gt;
Preliminary benchmark results using fio (posixaio, size=4G, end_fsync=1, iodepth=16, 60sec results) are below. 1M = 2^20.&lt;br /&gt;
&lt;br /&gt;
* Large block (1M) sequential read&lt;br /&gt;
** 1 thread: 1520MBps&lt;br /&gt;
** 2 threads: 2260MBps&lt;br /&gt;
** 4 threads: 4835MBps&lt;br /&gt;
** 8 threads: 9017MBps&lt;br /&gt;
* Large block random read&lt;br /&gt;
** 1 thread: 1020MBps&lt;br /&gt;
** 2 threads: 2020MBps&lt;br /&gt;
** 4 threads: 3790MBps&lt;br /&gt;
* Medium block (64K) sequential read&lt;br /&gt;
** 1 thread: 1220MBps&lt;br /&gt;
** 2 threads: 2141MBps&lt;br /&gt;
** 4 threads: 4508MBps&lt;br /&gt;
* Medium block random read&lt;br /&gt;
** 1 thread: 370MBps&lt;br /&gt;
** 2 threads: 883MBps&lt;br /&gt;
** 4 threads: 1735MBps&lt;br /&gt;
* Small block (4K) sequential read&lt;br /&gt;
** 1 thread: &lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
* Small block random read&lt;br /&gt;
** 1 thread: 42MBps&lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
&lt;br /&gt;
* Large block 75R/25W mixed IO:&lt;br /&gt;
** 1 thread: 889/297MBps&lt;br /&gt;
** 2 threads: 1567/525MBps&lt;br /&gt;
** 4 threads: 2620/879MBps&lt;br /&gt;
* Medium block 75R/25W mixed IO:&lt;br /&gt;
** 1 thread: 449/151MBps&lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
* Small block 75R/25W mixed IO:&lt;br /&gt;
** 1 thread: &lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
== Upstairs storage ==&lt;br /&gt;
&lt;br /&gt;
Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.&lt;br /&gt;
&lt;br /&gt;
This storage, and package tree, is entirely separate from main storage.&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Service:storage&amp;diff=3491</id>
		<title>Service:storage</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Service:storage&amp;diff=3491"/>
		<updated>2023-11-30T01:18:42Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Main storage: /home/users */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;[[Category:Service]]&lt;br /&gt;
&lt;br /&gt;
OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.&lt;br /&gt;
&lt;br /&gt;
Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.&lt;br /&gt;
&lt;br /&gt;
== Main storage: /home/users ==&lt;br /&gt;
&lt;br /&gt;
All systems in the CC utilize the main home directory NAS server on [[Infrastructure:mnemosyne | Mnemosyne]]. This is a primary storage engine with 12 15TB sas-3 SSDs in a raidz2 redundant storage array providing 128TB of disk in one volume.&lt;br /&gt;
&lt;br /&gt;
Single-thread IO tests yield ~1.8GBps sequential read and ~10K IOPS 4K random r/w.&lt;br /&gt;
&lt;br /&gt;
== Package tree: /packages ==&lt;br /&gt;
&lt;br /&gt;
Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1.&lt;br /&gt;
&lt;br /&gt;
As packages are subject to almost exclusively read access, they are mounted synchronously.&lt;br /&gt;
&lt;br /&gt;
The SSDs provide 1.2GBps sequential speed.&lt;br /&gt;
&lt;br /&gt;
== High performance storage: /gpfs/gpfs0 ==&lt;br /&gt;
&lt;br /&gt;
OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.&lt;br /&gt;
&lt;br /&gt;
The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.&lt;br /&gt;
&lt;br /&gt;
== High performance scratch: /storage/warpspeed ==&lt;br /&gt;
&lt;br /&gt;
Mnemosyne hosts a low-latency 8TB scratch volume made from two 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0.&lt;br /&gt;
&lt;br /&gt;
== BeeGFS storage ==&lt;br /&gt;
&lt;br /&gt;
Experimental deployment of a BeeGFS high-performance NFS is underway.&lt;br /&gt;
&lt;br /&gt;
The backing store is a ZFS pool to which are assigned:&lt;br /&gt;
* 2 mirrored 4T NVME drives for ZFS metadata&lt;br /&gt;
* 2 4T NVMEs assigned as cache&lt;br /&gt;
* 12 x 18T disks forming a 150TB volume in RaidZ2&lt;br /&gt;
&lt;br /&gt;
BeeGFS itself has been temporarily assigned a directory on the warpspeed filesystem (nvme mdadm raid1) for its own metadata.&lt;br /&gt;
&lt;br /&gt;
Preliminary benchmark results using fio (posixaio, size=4G, end_fsync=1, iodepth=16, 60sec results) are below. 1M = 2^20.&lt;br /&gt;
&lt;br /&gt;
* Large block (1M) sequential read&lt;br /&gt;
** 1 thread: 1520MBps&lt;br /&gt;
** 2 threads: 2260MBps&lt;br /&gt;
** 4 threads: 4835MBps&lt;br /&gt;
** 8 threads: 9017MBps&lt;br /&gt;
* Large block random read&lt;br /&gt;
** 1 thread: 1020MBps&lt;br /&gt;
** 2 threads: 2020MBps&lt;br /&gt;
** 4 threads: 3790MBps&lt;br /&gt;
* Medium block (64K) sequential read&lt;br /&gt;
** 1 thread: 1220MBps&lt;br /&gt;
** 2 threads: 2141MBps&lt;br /&gt;
** 4 threads: 4508MBps&lt;br /&gt;
* Medium block random read&lt;br /&gt;
** 1 thread: 370MBps&lt;br /&gt;
** 2 threads: 883MBps&lt;br /&gt;
** 4 threads: 1735MBps&lt;br /&gt;
* Small block (4K) sequential read&lt;br /&gt;
** 1 thread: &lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
* Small block random read&lt;br /&gt;
** 1 thread: 42MBps&lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
&lt;br /&gt;
* Large block 75R/25W mixed IO:&lt;br /&gt;
** 1 thread: 889/297MBps&lt;br /&gt;
** 2 threads: 1567/525MBps&lt;br /&gt;
** 4 threads: 2620/879MBps&lt;br /&gt;
* Medium block 75R/25W mixed IO:&lt;br /&gt;
** 1 thread: 449/151MBps&lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
* Small block 75R/25W mixed IO:&lt;br /&gt;
** 1 thread: &lt;br /&gt;
** 2 threads: &lt;br /&gt;
** 4 threads: &lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
== Upstairs storage ==&lt;br /&gt;
&lt;br /&gt;
Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.&lt;br /&gt;
&lt;br /&gt;
This storage, and package tree, is entirely separate from main storage.&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3358</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3358"/>
		<updated>2022-04-13T02:50:30Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Godzilla doesn't have two K80s. It has RHEL 8.5 instead of 8.2 now.&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 || RHEL 8.5 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Mothra]] || Intel + 4 x NVIDIA M40 || RHEL 8.5 || Cascade Lake GPU server || 2 x 24c Xeon Gold 6248R @ 3.0GHz || 10GbE || R84.U3&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3357</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3357"/>
		<updated>2022-03-24T04:48:56Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Cumulus and Nimbus have 10GbE&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Mothra]] || Intel + 4 x NVIDIA M40 || RHEL 8.5 || Cascade Lake GPU server || 2 x 24c Xeon Gold 6248R @ 3.0GHz || 10GbE || R84.U3&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3356</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3356"/>
		<updated>2022-03-24T04:47:38Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Deleted SX-Aurora&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Mothra]] || Intel + 4 x NVIDIA M40 || RHEL 8.5 || Cascade Lake GPU server || 2 x 24c Xeon Gold 6248R @ 3.0GHz || 10GbE || R84.U3&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3355</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3355"/>
		<updated>2022-03-15T17:34:21Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Mothra]] || Intel + 4 x NVIDIA M40 || RHEL 8.5 || Cascade Lake GPU server || 2 x 24c Xeon Gold 6248R @ 3.0GHz || 10GbE || R84.U3&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3354</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3354"/>
		<updated>2022-03-15T17:33:43Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Mothra. Needs rack location.&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Mothra]] || Intel + 4 x NVIDIA M40 || RHEL 8.5 || Cascade Lake GPU server || 2 x 24c Xeon Gold 6248R @ 3.0GHz || 10GbE || R84.U4&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3353</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3353"/>
		<updated>2022-03-15T02:43:29Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Mothra&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Mothra]] || Intel + 4 x NVIDIA M40 GPU || RHEL 8.5 || Cascade Lake GPU server || 2 x 24c Xeon Gold 6248R @ 3.0GHz || 10GbE || R86&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3352</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3352"/>
		<updated>2022-03-15T02:41:32Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Mothra. Needs rack location. R86?&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Mothra]] || Intel + 4 x M40 GPU node || RHEL 8.5 || Cascade Lake GPU server || 2 x 24c Xeon Gold 6248R @ 3.0GHz || 10GbE || R86&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3351</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3351"/>
		<updated>2022-03-05T00:26:24Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Ice lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Saturn&amp;diff=3350</id>
		<title>Compute: Saturn</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Saturn&amp;diff=3350"/>
		<updated>2022-03-05T00:26:13Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Hardware Info */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;== Hardware Info ==&lt;br /&gt;
&lt;br /&gt;
* CPU: 4 x Xeon 8367HC, 26 core Ice Lake @ 3.2GHz&lt;br /&gt;
* RAM: 384GB DDR4-3200 + 2.5TB Optane&lt;br /&gt;
* Disk: 900GB NVME (450G /, 450G docker)&lt;br /&gt;
* Ethernet:&lt;br /&gt;
** 10GbE public&lt;br /&gt;
** 100GbE private, P2100G&lt;br /&gt;
* Other NIC:&lt;br /&gt;
** Bluefield 2 ConnectX-6 DPU&lt;br /&gt;
* Accelerators:&lt;br /&gt;
** nVidia A100-80G&lt;br /&gt;
&lt;br /&gt;
== Software info ==&lt;br /&gt;
&lt;br /&gt;
* OS base: Ubuntu-20&lt;br /&gt;
* Drivers:&lt;br /&gt;
** nVidia - 470&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3349</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3349"/>
		<updated>2022-02-05T01:32:42Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Ubuntu on Instinct.&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Ubuntu 20.04.2 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Reptar&amp;diff=3348</id>
		<title>Compute: Reptar</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Reptar&amp;diff=3348"/>
		<updated>2022-02-05T01:30:39Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;== Hardware Info ==&lt;br /&gt;
&lt;br /&gt;
Reptar is a new hardware twin of Voltar (X11DPG-QT motherboard) but with even more powerful CPUs.&lt;br /&gt;
&lt;br /&gt;
Reptar is the first OACISS machine to have ''all three'' major accelerators: AMD, nVidia and Intel.&lt;br /&gt;
&lt;br /&gt;
* CPU: 2 x Xeon 6248R (24 core Cascade Lake @ 3.0GHz)&lt;br /&gt;
* RAM: 256GB DDR4-2933&lt;br /&gt;
* Disk: &lt;br /&gt;
** 570G SSD /&lt;br /&gt;
** 590G NVME docker&lt;br /&gt;
** 330G NVME /scratch&lt;br /&gt;
* Ethernet:&lt;br /&gt;
** 10GbE public&lt;br /&gt;
** 100GbE private&lt;br /&gt;
* Other NIC:&lt;br /&gt;
* Accelerators:&lt;br /&gt;
** AMD MI-50&lt;br /&gt;
** nVidia K80&lt;br /&gt;
&lt;br /&gt;
== Software info ==&lt;br /&gt;
&lt;br /&gt;
* OS base: RHEL-8.5&lt;br /&gt;
* Drivers:&lt;br /&gt;
** nVidia - 450.51&lt;br /&gt;
* Other software&lt;br /&gt;
** ROCm&lt;br /&gt;
** Intel OneAPI&lt;br /&gt;
** CUDA-11.0 [last version that supports sm_37]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3347</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3347"/>
		<updated>2022-02-05T01:29:13Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || 2x6248R CPU + AMD + nVidia || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Echs&amp;diff=3346</id>
		<title>Compute: Echs</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Echs&amp;diff=3346"/>
		<updated>2022-02-05T01:28:11Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Created page with &amp;quot;== Hardware Info ==  * CPU: i7-10700 CPU @ 2.90GHz * RAM: 128GB DDR4-2666 * Disk: 500G SSD * Ethernet: 1G uplink * Other NIC: * Accelerators:  == Software info ==  * OS base:...&amp;quot;&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;== Hardware Info ==&lt;br /&gt;
&lt;br /&gt;
* CPU: i7-10700 CPU @ 2.90GHz&lt;br /&gt;
* RAM: 128GB DDR4-2666&lt;br /&gt;
* Disk: 500G SSD&lt;br /&gt;
* Ethernet: 1G uplink&lt;br /&gt;
* Other NIC:&lt;br /&gt;
* Accelerators:&lt;br /&gt;
&lt;br /&gt;
== Software info ==&lt;br /&gt;
&lt;br /&gt;
* OS base: Ubuntu 20&lt;br /&gt;
* Drivers:&lt;br /&gt;
** nVidia - &lt;br /&gt;
* Other software&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3345</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3345"/>
		<updated>2022-02-05T01:25:01Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Echs]] || Intel box || Ubuntu 20 || MSI X590 || 8 core i7-10700 || 2.5GbE || Str-470 foyer &lt;br /&gt;
|- &lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:New_LDAP_Client&amp;diff=3344</id>
		<title>Procedure:New LDAP Client</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:New_LDAP_Client&amp;diff=3344"/>
		<updated>2022-02-04T23:23:51Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Replace /etc/ldap.conf */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page documents the steps necessary to add a new client that will use LDAP accounts and mount the shared home directories over NFS.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
== Host Setup Steps ==&lt;br /&gt;
&lt;br /&gt;
These steps are valid for Red Hat 4.x and 5.x systems, Red Hat and derivatives such as CentOS, Scientific Linux, and Rocks.&lt;br /&gt;
* For other Linux distributions, follow the AutoFS sections of this page, as well as the [[Procedures:New_LDAP_Client/Manual_Setup|Manual Setup steps]].&lt;br /&gt;
* For Solaris , see [[Procedures:New_LDAP_Client/Solaris_10_Setup|Solaris 10 steps]].&lt;br /&gt;
* For AIX 5.3, see [[Procedures:New_LDAP_Client/AIX_5.3_Setup|AIX 5.3 Setup steps]].&lt;br /&gt;
* AIX 5.2 and earlier hosts do not support OpenLDAP, but can still mount NFS. See the [[Procedures:New_LDAP_Client/AIX_5.2_Setup|AIX 5.2 Setup steps]], and the [[Procedures:New_User/AIX|AIX User Creation Proceedure]] for user synchronization steps.&lt;br /&gt;
&lt;br /&gt;
== Prerequisites ==&lt;br /&gt;
&lt;br /&gt;
Make sure the following requirements are met on the new system:&lt;br /&gt;
&lt;br /&gt;
* Packages&lt;br /&gt;
** autofs&lt;br /&gt;
** pam_ldap&lt;br /&gt;
** nss_ldap&lt;br /&gt;
** authconfig&lt;br /&gt;
** openldap-clients&lt;br /&gt;
* Network&lt;br /&gt;
** On Storage Network (172.17.x.x IP address)&lt;br /&gt;
** 9000 byte MTU Jumbo frames (for best performance; [[Procedures:New_LDAP_Client#Create_Netapp_AutoFS_map_file|see note below]])&lt;br /&gt;
&lt;br /&gt;
=== Replace /etc/ldap.conf ===&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
cat &amp;lt;&amp;lt;EOF &amp;gt;/etc/ldap.conf&lt;br /&gt;
binddn cn=anonymous,dc=nic,dc=uoregon,dc=edu&lt;br /&gt;
bindpw ________&lt;br /&gt;
pam_check_host_attr yes&lt;br /&gt;
scope sub&lt;br /&gt;
nss_base_passwd ou=people,dc=nic,dc=uoregon,dc=edu?one&lt;br /&gt;
nss_base_shadow ou=people,dc=nic,dc=uoregon,dc=edu?one&lt;br /&gt;
nss_base_group  ou=group,dc=nic,dc=uoregon,dc=edu?one&lt;br /&gt;
tls_checkpeer no&lt;br /&gt;
EOF&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Replace /etc/openldap/ldap.conf ===&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
cat &amp;lt;&amp;lt;EOF &amp;gt;/etc/openldap/ldap.conf&lt;br /&gt;
TLS_CHECKPEER no&lt;br /&gt;
EOF&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Create Netapp AutoFS map file ===&lt;br /&gt;
''' Note:''' If your host does not support jumbo frames, replace 'udp' in these lines with 'tcp', or filesystem operations on NFS paths may fail.&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
cat &amp;lt;&amp;lt;EOF &amp;gt;/etc/auto.netapp&lt;br /&gt;
home       -fstype=nfs,hard,intr,async,rsize=32768,wsize=32768,nfsvers=3,udp     172.17.8.63:/vol/home&lt;br /&gt;
home1      -fstype=nfs,hard,intr,async,rsize=32768,wsize=32768,nfsvers=3,udp     172.17.8.63:/vol/home1&lt;br /&gt;
home2      -fstype=nfs,hard,intr,async,rsize=32768,wsize=32768,nfsvers=3,udp     172.17.8.64:/vol/home2&lt;br /&gt;
research   -fstype=nfs,hard,intr,async,rsize=32768,wsize=32768,nfsvers=3,udp     172.17.8.64:/vol/research&lt;br /&gt;
packages   -fstype=nfs,hard,intr,async,rsize=32768,wsize=32768,nfsvers=3,udp     172.17.8.64:/vol/packages/ARCH&lt;br /&gt;
EOF&lt;br /&gt;
sed -i -e &amp;quot;s/ARCH/`uname -i`/&amp;quot; /etc/auto.netapp&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Append Netapp maps to AutoFS master map ===&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
cat &amp;lt;&amp;lt;EOF &amp;gt;&amp;gt;/etc/auto.master&lt;br /&gt;
/mnt/netapp     /etc/auto.netapp        --timeout=1200 --ghost&lt;br /&gt;
EOF&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Enable Netapp mounts and add path links ===&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
service autofs reload&lt;br /&gt;
ln -sf /mnt/netapp/packages/ /usr/local/packages&lt;br /&gt;
ln -sf /mnt/netapp/home/users/ /home/users&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Enable LDAP authentication ===&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
authconfig --update --enablecache --enablepamaccess \&lt;br /&gt;
           --enableldap --enableldapauth --enableldapssl \&lt;br /&gt;
           --ldapbasedn=dc=nic,dc=uoregon,dc=edu \&lt;br /&gt;
           --ldaploadcacert=http://systems.nic.uoregon.edu/ca/NIC-cacert.pem \&lt;br /&gt;
           --ldapserver=172.17.202.25,172.17.8.66&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Test Configuration ===&lt;br /&gt;
If everything is set up right, you should be able to run the following commands, and see some output without any errors or hanging:&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
getent passwd chemadmin&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
ls -la /mnt/netapp/home&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== LDAP Setup Steps ==&lt;br /&gt;
In order to enable granular host access control, you must also log into the [https://systems.nic.uoregon.edu/slam/ NIC user control panel], go to the 'Manage Hosts' section, and add an entry for the new system.&lt;br /&gt;
&lt;br /&gt;
The 'Host FQDN' field should match the results of the 'hostname' command when run on the new host. The 'Display Name' field is optional, and is only used when informing the user what hosts they are allowed to log in to, in place of the FQDN. After the host has been added, you may select the entry, click the 'Edit Selected' button, and select which users are allowed to log into the host.&lt;br /&gt;
&lt;br /&gt;
'''Note:''' The functionality of this feature depends on the host's LDAP libraries honoring the 'pam_check_host_attr' option. If it does not, any user with an LDAP account will be able to log in. &lt;br /&gt;
&lt;br /&gt;
[[Category:Proceedure]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3343</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3343"/>
		<updated>2022-02-03T09:40:16Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Test program */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI, and the exact same MPI runtime to execute it&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;--map-by node&amp;lt;/code&amp;gt; is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.&lt;br /&gt;
&lt;br /&gt;
The use of &amp;lt;code&amp;gt;--prefix&amp;lt;/code&amp;gt; is key as, without using a cluster scheduler, there is no other way to invoke the correct &amp;lt;code&amp;gt;orted&amp;lt;/code&amp;gt; and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,&lt;br /&gt;
&amp;lt;code&amp;gt; --prefix $(which orted | sed -e 's/\/bin\/orted//')&amp;lt;/code&amp;gt; can be used to do this inline.&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Running on the 8-node AXIS cluster ===&lt;br /&gt;
&lt;br /&gt;
OACISS has a small test cluster designated AXIS run by SLURM. This cluster is visible from the orthus login node via the standard slurm commands (sinfo/squeue/srun/salloc/etc).&lt;br /&gt;
&lt;br /&gt;
By default, when an environment is setup inside a Slurm script, Slurm copies that environment for the invoked processes and this is helpful. However because of differences in the installation of Slurm on the Axis nodes and on Orthus, this causes 'srun -N [2 or more] ./foo.sh' to fail when invoked from Orthus. Thus, successful of use of srun on the axis cluster must be performed from one of the axis nodes.&lt;br /&gt;
&lt;br /&gt;
The nodes are axis1 through axis8. They live only on the private network and share a 10 gigabit interconnect.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-llvm8.0.1 ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3342</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3342"/>
		<updated>2022-02-03T09:34:53Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* General forewords */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI, and the exact same MPI runtime to execute it&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;--map-by node&amp;lt;/code&amp;gt; is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.&lt;br /&gt;
&lt;br /&gt;
The use of &amp;lt;code&amp;gt;--prefix&amp;lt;/code&amp;gt; is key as, without using a cluster scheduler, there is no other way to invoke the correct &amp;lt;pre&amp;gt;orted&amp;lt;/pre&amp;gt; and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,&lt;br /&gt;
&amp;lt;code&amp;gt; --prefix $(which orted | sed -e 's/\/bin\/orted//')&amp;lt;/code&amp;gt; can be used to do this inline.&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Running on the 8-node AXIS cluster ===&lt;br /&gt;
&lt;br /&gt;
OACISS has a small test cluster designated AXIS run by SLURM. This cluster is visible from the orthus login node via the standard slurm commands (sinfo/squeue/srun/salloc/etc).&lt;br /&gt;
&lt;br /&gt;
By default, when an environment is setup inside a Slurm script, Slurm copies that environment for the invoked processes and this is helpful. However because of differences in the installation of Slurm on the Axis nodes and on Orthus, this causes 'srun -N [2 or more] ./foo.sh' to fail when invoked from Orthus. Thus, successful of use of srun on the axis cluster must be performed from one of the axis nodes.&lt;br /&gt;
&lt;br /&gt;
The nodes are axis1 through axis8. They live only on the private network and share a 10 gigabit interconnect.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-llvm8.0.1 ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3341</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3341"/>
		<updated>2022-02-03T09:34:19Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;--map-by node&amp;lt;/code&amp;gt; is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.&lt;br /&gt;
&lt;br /&gt;
The use of &amp;lt;code&amp;gt;--prefix&amp;lt;/code&amp;gt; is key as, without using a cluster scheduler, there is no other way to invoke the correct &amp;lt;pre&amp;gt;orted&amp;lt;/pre&amp;gt; and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,&lt;br /&gt;
&amp;lt;code&amp;gt; --prefix $(which orted | sed -e 's/\/bin\/orted//')&amp;lt;/code&amp;gt; can be used to do this inline.&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Running on the 8-node AXIS cluster ===&lt;br /&gt;
&lt;br /&gt;
OACISS has a small test cluster designated AXIS run by SLURM. This cluster is visible from the orthus login node via the standard slurm commands (sinfo/squeue/srun/salloc/etc).&lt;br /&gt;
&lt;br /&gt;
By default, when an environment is setup inside a Slurm script, Slurm copies that environment for the invoked processes and this is helpful. However because of differences in the installation of Slurm on the Axis nodes and on Orthus, this causes 'srun -N [2 or more] ./foo.sh' to fail when invoked from Orthus. Thus, successful of use of srun on the axis cluster must be performed from one of the axis nodes.&lt;br /&gt;
&lt;br /&gt;
The nodes are axis1 through axis8. They live only on the private network and share a 10 gigabit interconnect.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-llvm8.0.1 ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3340</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3340"/>
		<updated>2022-02-03T09:27:23Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Test program */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
The &amp;lt;code&amp;gt;--map-by node&amp;lt;/code is needed because if it isn't present, MPI will observe that cyclops has enough cores for 6 ranks by itself and all ranks will run on cyclops.&lt;br /&gt;
&lt;br /&gt;
The use of &amp;lt;code&amp;gt;--prefix&amp;lt;/code&amp;gt; is key as, without using a cluster scheduler, there is no other way to invoke the correct &amp;lt;pre&amp;gt;orted&amp;lt;/pre&amp;gt; and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,&lt;br /&gt;
&amp;lt;code&amp;gt; --prefix $(which orted | sed -e 's/\/bin\/orted//')&amp;lt;/code&amp;gt; can be used to do this inline.&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-llvm8.0.1 ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3339</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3339"/>
		<updated>2022-02-03T09:19:03Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Test program */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
The use of --prefix is key as, without using a cluster scheduler, there is no other way to invoke the correct &amp;lt;pre&amp;gt;orted&amp;lt;/pre&amp;gt; and MPI will fail with a message to the effect of being unable to find it. The simplest way to find it is to use 'which orted' with the MPI module you want to use, which will report a full path, and then remove /bin/orted. Or,&lt;br /&gt;
&amp;lt;code&amp;gt; --prefix $(which orted | sed -e 's/\/bin\/orted//')&amp;lt;/code&amp;gt; can be used to do this inline.&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-llvm8.0.1 ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3338</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3338"/>
		<updated>2022-02-03T02:05:36Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-llvm8.0.1 ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3337</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3337"/>
		<updated>2022-02-03T02:05:15Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ vi hostlist&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-llvm8.0.1 ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) llvm/8.0.1   2) openmpi/4.0.1-llvm8.0.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-llvm8.0.1 --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
 (...)&lt;br /&gt;
erik-k@cyclops ~/mpitester $ echo $?&lt;br /&gt;
0&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3336</id>
		<title>HowtoMPI</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HowtoMPI&amp;diff=3336"/>
		<updated>2022-02-03T01:58:40Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Created page with &amp;quot;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.  == General forewords ==  There are a few key factors that hav...&amp;quot;&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page describes a number of different tested-working combinations of MPI that are usable on OACISS hardware.&lt;br /&gt;
&lt;br /&gt;
== General forewords ==&lt;br /&gt;
&lt;br /&gt;
There are a few key factors that have to be considered in general to get MPI to work. First is that major MPI implementations (Mpich, OpenMpi, Spectrum Mpi) do a huge amount of low-level tuning, which leads to MPI being a famously delicate snowflake when it comes to the API and ABI. This leads to the main warning:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;Thou shalt use the exact same compiler to compile and link thy code to MPI that compiled MPI&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is assisted in most cases in OACISS environments by the fact that most MPI modules automatically load the compiler that built them as well, for this reason.&lt;br /&gt;
&lt;br /&gt;
Another concern specific to OACISS' highly heterogeneous environment is that we have multiple entire package trees for different operating systems and even processor architectures. We expect it would be very difficult to get MPI to run across Cascade Lake Xeon systems and Power9 systems, and quite difficult to achieve software compatibility across wholly different operating systems (Currently, we have RHEL7.9, RHEL8.x, Ubuntu 20 and AIX 7.2 nodes available).&lt;br /&gt;
&lt;br /&gt;
== Test program ==&lt;br /&gt;
&lt;br /&gt;
This ring.c test does the classic MPI &amp;quot;print my rank and host names&amp;quot; helloworld and adds a minimal nontrivial amount of communication which requires that MPI actually succeed at passing messages (even if this pass-the-token demo also represents an extreme example of a parallel program which has no concurrency at all),&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#include &amp;lt;mpi.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdio.h&amp;gt;&lt;br /&gt;
#include &amp;lt;stdlib.h&amp;gt;&lt;br /&gt;
&lt;br /&gt;
void ringSend(int me, int proc) {&lt;br /&gt;
  int i;&lt;br /&gt;
  int field = -1;&lt;br /&gt;
  MPI_Status status;&lt;br /&gt;
&lt;br /&gt;
  MPI_Barrier(MPI_COMM_WORLD);&lt;br /&gt;
&lt;br /&gt;
  int fieldTx;&lt;br /&gt;
&lt;br /&gt;
  char name[MPI_MAX_PROCESSOR_NAME];&lt;br /&gt;
  int namelen;&lt;br /&gt;
  MPI_Get_processor_name(name, &amp;amp;namelen);&lt;br /&gt;
&lt;br /&gt;
  if (me==0) {&lt;br /&gt;
    field = 0;&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, 1, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, proc-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
  else {&lt;br /&gt;
    MPI_Recv(&amp;amp;field, 1, MPI_INT, me-1, 4711, MPI_COMM_WORLD, &amp;amp;status);&lt;br /&gt;
    fieldTx = field + 1;&lt;br /&gt;
    MPI_Send(&amp;amp;fieldTx, 1, MPI_INT, (me+1)%proc, 4711, MPI_COMM_WORLD);&lt;br /&gt;
    printf(&amp;quot;Rank %d on %s: field rx = %d, field tx = %d\n&amp;quot;, me, name, field, fieldTx);&lt;br /&gt;
  }&lt;br /&gt;
&lt;br /&gt;
}&lt;br /&gt;
&lt;br /&gt;
int main(int argc, char **argv) {&lt;br /&gt;
  int proc, me;&lt;br /&gt;
&lt;br /&gt;
  MPI_Init (&amp;amp;argc, &amp;amp; argv);&lt;br /&gt;
  MPI_Comm_size (MPI_COMM_WORLD, &amp;amp;proc);&lt;br /&gt;
  MPI_Comm_rank (MPI_COMM_WORLD, &amp;amp;me);&lt;br /&gt;
&lt;br /&gt;
  ringSend(me, proc);&lt;br /&gt;
&lt;br /&gt;
  MPI_Finalize ();&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
An example output, excepting the usual complaints from openmpi, might look like&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
WARNING: No preset parameters ...&lt;br /&gt;
... (blah blah blah) ...&lt;br /&gt;
--------------------------------------------------------------------------&lt;br /&gt;
1 on gorgon.stor: field rx = 1, field tx = 2&lt;br /&gt;
3 on gorgon.stor: field rx = 3, field tx = 4&lt;br /&gt;
5 on gorgon.stor: field rx = 5, field tx = 6&lt;br /&gt;
0 on cyclops.stor: field rx = 6, field tx = 1&lt;br /&gt;
2 on cyclops.stor: field rx = 2, field tx = 3&lt;br /&gt;
4 on cyclops.stor: field rx = 4, field tx = 5&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As you can see by reading the code, each rank N receives an integer from rank N-1, increments it by 1, and passes it to rank N+1. Above, the loop completes when rank 0 receives an integer from rank 5, which has by then been incremented 6 times.&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
== MPI communication ==&lt;br /&gt;
&lt;br /&gt;
For those interested in MPI message passing performance, a word about MPI behaviors is in order. MPI will detect if communicating ranks are on a single node, and communication between them will proceed through shared memory without ever invoking an actual tcp socket.&lt;br /&gt;
&lt;br /&gt;
OpenMPI will also (by default) promiscuously detect and use all available Ethernet network interfaces, bonding them at the software level to aggregate their bandwidth. Nearly all OACISS systems have two network interfaces and MPI will not only use both, but will usually fail if any two ranks are unable to communicate using any interface. This behavior can be changed using &amp;lt;pre&amp;gt;-mca btl tcp_if_include=x&amp;lt;/pre&amp;gt; or &amp;lt;pre&amp;gt;-mca btl tcp_if_exclude=x&amp;lt;/pre&amp;gt; arguments.&lt;br /&gt;
&lt;br /&gt;
There is not a known (to us) solution to this problem on heterogeneous nodes whose hardware interface names differ.&lt;br /&gt;
&lt;br /&gt;
== Tested environments/combinations ==&lt;br /&gt;
&lt;br /&gt;
Unless otherwise stated, the below examples are presumptively using the above ring.c program. Success is assumed &amp;amp; the actual output from mpirun is not included.&lt;br /&gt;
&lt;br /&gt;
=== Power9 / Openmpi-4.0.1-gcc10.1 module (Feb 2 2022) ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k@cyclops ~/mpitester $ module list&lt;br /&gt;
Currently Loaded Modules:&lt;br /&gt;
  1) gcc/10.1   2) openmpi/4.0.1-gcc10.1&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpicc -o ring ring.c&lt;br /&gt;
erik-k@cyclops ~/mpitester $ cat hostlist&lt;br /&gt;
cyclops&lt;br /&gt;
gorgon&lt;br /&gt;
erik-k@cyclops ~/mpitester $ vi hostlist&lt;br /&gt;
erik-k@cyclops ~/mpitester $ mpirun --prefix /packages/openmpi/4.0.1-gcc10.1/ --hostfile hostlist -np 6 --map-by node ./ring&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Procedures&amp;diff=3335</id>
		<title>Category:Procedures</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Procedures&amp;diff=3335"/>
		<updated>2022-02-03T01:19:21Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;Instructions on how to accomplish various common tasks:&lt;br /&gt;
&lt;br /&gt;
* [[Procedure:AIX_jumbo_frames|AIX jumbo frames - how to enable jumbo frames on a compatible ethernet adapter]]&lt;br /&gt;
* [[Infrastructure:Lights-Out_Management|Lights-out management - how to control a host remotely]]&lt;br /&gt;
* [[Procedure:Chymera X11 setup]]&lt;br /&gt;
* [[Procedure:MySQL Database Transfers]]&lt;br /&gt;
* [[Procedure:New Host / Reinstallation setup]]&lt;br /&gt;
* [[Procedure:New LDAP Client|Setting up a new machine as an LDAP/NFS client]]&lt;br /&gt;
* [[Procedure:New host filesystems]]&lt;br /&gt;
* [[Procedure:New User|Adding a new user]]&lt;br /&gt;
* [[Procedure:NIM Setup|NIM Setup - how to go from fresh AIX install to CSM-integrated NIM Master]]&lt;br /&gt;
* [[Procedure:Perfctr|Creating a Perfctr enabled (PAPI) kernel rpm]]&lt;br /&gt;
* [[Procedure:Reserving a Conference Room]]&lt;br /&gt;
* [[Procedure:Set up PostgreSQL database]]&lt;br /&gt;
* [[Procedure:Setting up and Accessing ISCSI Volumes]]&lt;br /&gt;
* [[Procedure:WebServicesMigration]]&lt;br /&gt;
* [[Procedure:WinXP VM]]&lt;br /&gt;
* [[Procedure:MPSS_setup]]&lt;br /&gt;
* [[Procedure:Setting up update proxy]]&lt;br /&gt;
* [[Procedure:SpectrumScaleInstall]]&lt;br /&gt;
* [[Procedure:HostDiskMigration]]&lt;br /&gt;
&lt;br /&gt;
* [[HowtoMPI]] describes various tested-working MPI setups&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3334</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3334"/>
		<updated>2022-02-03T01:18:59Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
Some relevant pages:&lt;br /&gt;
* The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
* The [[Service:storage]] describes available storage for users of OACISS systems. OACISS currently has a total of just under 350TB of online storage available.&lt;br /&gt;
* The new [[HowtoMPI]] page describes various tested-working MPI setups and the steps&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3333</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3333"/>
		<updated>2022-01-20T21:03:57Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Axis&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
&lt;br /&gt;
The [[Service:storage]] describes available storage for users of OACISS systems.&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
OACISS has a large amount of storage available: See [[Service:storage | Storage]].&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster (axis1-8)]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes with Slurm || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3332</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3332"/>
		<updated>2022-01-12T19:07:33Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: DG1 moved from Reptar to Godzilla&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
&lt;br /&gt;
The [[Service:storage]] describes available storage for users of OACISS systems.&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
OACISS has a large amount of storage available: See [[Service:storage | Storage]].&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node || RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel DG1 + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3331</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3331"/>
		<updated>2022-01-05T23:16:39Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
&lt;br /&gt;
The [[Service:storage]] describes available storage for users of OACISS systems.&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
OACISS has a large amount of storage available: See [[Service:storage | Storage]].&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Nodes]] in Computing Center datacenter&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node + Intel DG1|| RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Axis_cluster&amp;diff=3330</id>
		<title>Compute: Axis cluster</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Axis_cluster&amp;diff=3330"/>
		<updated>2022-01-05T04:57:18Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Hardware Info */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;== Hardware Info ==&lt;br /&gt;
&lt;br /&gt;
* CPU: 4 x 8 core Nehalem Xeon, 2.27GHz&lt;br /&gt;
* RAM: 512GB ddr3-1066&lt;br /&gt;
* Disk:&lt;br /&gt;
** 500GB rotating local disk&lt;br /&gt;
** 140TB home directory&lt;br /&gt;
** 200TB flash /gpfs&lt;br /&gt;
* Ethernet: 10GbE&lt;br /&gt;
* Other NIC:&lt;br /&gt;
* Accelerators:&lt;br /&gt;
&lt;br /&gt;
== Software info ==&lt;br /&gt;
&lt;br /&gt;
* OS base: RHEL 8.5&lt;br /&gt;
* Drivers:&lt;br /&gt;
** nVidia - &lt;br /&gt;
* Other software&lt;br /&gt;
** Local package tree of compilers &amp;amp; MPI runtimes&lt;br /&gt;
&lt;br /&gt;
Interactive access from Orthus: &amp;lt;pre&amp;gt;srun --pty bash&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Axis_cluster&amp;diff=3329</id>
		<title>Compute: Axis cluster</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Compute:_Axis_cluster&amp;diff=3329"/>
		<updated>2022-01-05T04:54:56Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Created page with &amp;quot;== Hardware Info ==  * CPU: 4 x 8 core Nehalem Xeon, 2.27GHz * RAM: 512GB ddr3-1066 * Disk: 500GB rotating local disk * Ethernet: 10GbE * Other NIC: * Accelerators:  == Softwa...&amp;quot;&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;== Hardware Info ==&lt;br /&gt;
&lt;br /&gt;
* CPU: 4 x 8 core Nehalem Xeon, 2.27GHz&lt;br /&gt;
* RAM: 512GB ddr3-1066&lt;br /&gt;
* Disk: 500GB rotating local disk&lt;br /&gt;
* Ethernet: 10GbE&lt;br /&gt;
* Other NIC:&lt;br /&gt;
* Accelerators:&lt;br /&gt;
&lt;br /&gt;
== Software info ==&lt;br /&gt;
&lt;br /&gt;
* OS base: RHEL 8.5&lt;br /&gt;
* Drivers:&lt;br /&gt;
** nVidia - &lt;br /&gt;
* Other software&lt;br /&gt;
** Local package tree of compilers &amp;amp; MPI runtimes&lt;br /&gt;
&lt;br /&gt;
Interactive access from Orthus: &amp;lt;pre&amp;gt;srun --pty bash&amp;lt;/pre&amp;gt;&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3328</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3328"/>
		<updated>2022-01-05T04:53:10Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
&lt;br /&gt;
The [[Service:storage]] describes available storage for users of OACISS systems.&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
OACISS has a large amount of storage available: See [[Service:storage | Storage]].&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Deschutes machine room&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node + Intel DG1|| RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Axis cluster]] || DL580 G7 nodes || RHEL 8.5 || HP 4U compute nodes || 4 x 8c Xeon Nehalem @ 2.3GHz || 10GbE || R82&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3327</id>
		<title>Procedure:OpenshiftInstall</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3327"/>
		<updated>2022-01-05T03:23:51Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Worker install on IBM s924 */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page documents the long, painful and complex process of installing OpenShift in a manner that will hopefully reveal the numerous rakes in the grass and avoid a repeat of the famous Sideshow Bob scene.&lt;br /&gt;
&lt;br /&gt;
The big picture:&lt;br /&gt;
&lt;br /&gt;
asdfasdfasdf&lt;br /&gt;
&lt;br /&gt;
= Outside guidance URLs =&lt;br /&gt;
&lt;br /&gt;
* The ostensible guide: https://docs.openshift.com/container-platform/4.6/installing/installing_bare_metal/installing-bare-metal.html#installation-obtaining-installer_installing-bare-metal&lt;br /&gt;
** This reads a lot like a semi-organized stream of consciousness, simply proceeding from A to B to C with no indexing and few explanations.&lt;br /&gt;
* https://cloud.redhat.com/openshift/downloads&lt;br /&gt;
** This leads to the openshift mirror repos where the openshift_client and openshift_installer tar files are found&lt;br /&gt;
* https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos&lt;br /&gt;
** '''PLEASE NOTE''': It is absolutely ''CRITICAL'' to have the SAME versions of the client, installer and rhcos&lt;br /&gt;
** Navigate specifically to version/xx, do not just grab 'stable'.&lt;br /&gt;
* https://www.redhat.com/en/blog/installing-openshift-41-using-libvirt-and-kvm&lt;br /&gt;
** This blog page provides some useful guidance and especially a useful template for the HAProxy configuration&lt;br /&gt;
&lt;br /&gt;
I note that this document describes the installation of a specific and isolated OC cluster.&lt;br /&gt;
&lt;br /&gt;
The OpenShift client system is itself a small VM named client.openshift.stor; This client also runs the proxy/load balancer.&lt;br /&gt;
&lt;br /&gt;
= External prerequisites =&lt;br /&gt;
&lt;br /&gt;
Openshift requires several services/components external to itself in order to work. These are,&lt;br /&gt;
* Web server: for serving up open RHCOS images and Ignition files to the installer&lt;br /&gt;
* Ethernet virbr: As this instance considers the installation of the entire cluster onto VMs living on a single real host, a virtual ethernet bridge must be established to facilitate communication&lt;br /&gt;
* Ethernet NAT masquerade: The cluster lives on a private LAN, and in this case the host machine is configured to provide NAT service to the OC virtual machines&lt;br /&gt;
* DNS: Openshift requires for itself a subdomain and a certain set of forward- and reverse-defined DNS entries.&lt;br /&gt;
* Load balancer: In this case, we install HAProxy on the client system and utilize it as the balancer. Alternatively, a dedicated balancer (VM, or machine, or hardware appliance) may be used.&lt;br /&gt;
&lt;br /&gt;
== Topology Foreword ==&lt;br /&gt;
&lt;br /&gt;
For clarity: The topology of connectivity utilized is&lt;br /&gt;
&lt;br /&gt;
(insert image here)&lt;br /&gt;
&lt;br /&gt;
== Ethernet connectivity ==&lt;br /&gt;
&lt;br /&gt;
The ethernet setup required takes two steps. First, a virtual ethernet bridge (software Level 2) must be setup on each VM host's private ethernet interface. This will create a br0 interface for the virtual bridge. The hardware en___ interface will lose its IP and be slaved to the bridge, and the br0 interface will acquire the IP.&lt;br /&gt;
&lt;br /&gt;
Now, qemu-kvm virtual machines will be able to connect to the br0 bridge, which (from their perspective) is as good as being on the physical Ethernet switch.&lt;br /&gt;
&lt;br /&gt;
As the second step, an IP masquerade/NAT is established for the cluster IPs such that they are able to name the host IP as their gateway and reach the wider Internet (to download updates and containers).&lt;br /&gt;
&lt;br /&gt;
=== Bridge setup ===&lt;br /&gt;
&lt;br /&gt;
Creating the virbr is simple on an Ubuntu host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;root@vina:~# cat /etc/netplan/00-installer-config.yaml &lt;br /&gt;
# This is the network config written by 'subiquity'&lt;br /&gt;
network:&lt;br /&gt;
  ethernets:&lt;br /&gt;
    enp1s0f0:&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
    enp1s0f1:&lt;br /&gt;
#      dhcp4: true&lt;br /&gt;
      mtu: 9000&lt;br /&gt;
  version: 2&lt;br /&gt;
  bridges:&lt;br /&gt;
    br0:&lt;br /&gt;
      interfaces: [enp1s0f1]&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
      mtu: 9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that here, mtu=9000 is set because our storage/private ethernet is setup to use jumbo frames and it is quite necessary that the IP MTUs match, or magic packet loss is likely to occur; Software bridges may automatically fragment jumbo frames, but hardware will simply discard overlength packets.&lt;br /&gt;
&lt;br /&gt;
The bridging setup on RHEL can be achieved several ways. I prefer to just punch it straight into /etc/sysconfig/network-scripts/*, as in this example that sets up one of the virbrs on our infrastructure vm node,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-eno1&lt;br /&gt;
TYPE=Ethernet&lt;br /&gt;
NAME=eno1-bridge-slave&lt;br /&gt;
UUID=9e7e3a89-3358-4830-a033-0a4154c68c55&lt;br /&gt;
DEVICE=eno1&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
BRIDGE=br0&lt;br /&gt;
HWADDR=3c:ec:ef:1a:71:3e&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-br0&lt;br /&gt;
STP=no&lt;br /&gt;
BRIDGING_OPTS=priority=32768&lt;br /&gt;
TYPE=Bridge&lt;br /&gt;
PROXY_METHOD=none&lt;br /&gt;
BROWSER_ONLY=no&lt;br /&gt;
BOOTPROTO=dhcp&lt;br /&gt;
DEFROUTE=yes&lt;br /&gt;
IPV4_FAILURE_FATAL=no&lt;br /&gt;
IPV6INIT=yes&lt;br /&gt;
IPV6_AUTOCONF=yes&lt;br /&gt;
IPV6_DEFROUTE=yes&lt;br /&gt;
IPV6_FAILURE_FATAL=no&lt;br /&gt;
IPV6_ADDR_GEN_MODE=stable-privacy&lt;br /&gt;
NAME=br0&lt;br /&gt;
UUID=3549a392-12ac-4c7f-bdd5-4e86d8654ddf&lt;br /&gt;
DEVICE=br0&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
MTU=9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It may be possible to leave STP enabled, however this was found to cause problems in some settings (the interface claims to be activating forever and never enters forwarding - i.e. the host physical ethernet port did not talk STP back)&lt;br /&gt;
&lt;br /&gt;
Once the bridge has been defined, we must inform kvm about it by defining an xml, importing it and marking it to autostart in kvm.&lt;br /&gt;
&lt;br /&gt;
=== IP masquerading ===&lt;br /&gt;
&lt;br /&gt;
Assuming that routing is enabled on the host and normal routing table entries are setup, the following three lines will setup IP masquerade between the public interface ('ifpublic') and br0,&lt;br /&gt;
&amp;lt;pre&amp;gt;iptables -A FORWARD -i br0 -o ifpublic -j ACCEPT&lt;br /&gt;
iptables -A FORWARD -i ifpublic -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT&lt;br /&gt;
iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.&lt;br /&gt;
&lt;br /&gt;
== Web server ==&lt;br /&gt;
&lt;br /&gt;
Some web space must be provisioned that can serve up&lt;br /&gt;
&lt;br /&gt;
* RHCOS images&lt;br /&gt;
* Ignition files&lt;br /&gt;
&lt;br /&gt;
The system at this stage is not paranoid; In our case, I have setup a 10-openshift.conf Apache server bound to Pliny's private network interface (172.17.202.12). Stripping all commentary from the configuration file,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:80&amp;gt;&lt;br /&gt;
    ServerAdmin systems@nic.uoregon.edu&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    DocumentRoot /home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
    # if not specified, the global error log is used&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/error_log&lt;br /&gt;
    CustomLog /var/log/httpd/openshift/access_log combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        options +indexes&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:443&amp;gt;&lt;br /&gt;
    #  General setup for the virtual host&lt;br /&gt;
    DocumentRoot &amp;quot;/home/web_openshift/&amp;quot;&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/ssl_error_log&lt;br /&gt;
    TransferLog /var/log/httpd/openshift/ssl_access_log&lt;br /&gt;
&lt;br /&gt;
    SSLEngine on&lt;br /&gt;
&lt;br /&gt;
    #   SSL Protocol Support:&lt;br /&gt;
    SSLProtocol All -SSLv2 -SSLv3&lt;br /&gt;
    SSLCipherSuite    ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA&lt;br /&gt;
    SSLHonorCipherOrder     on&lt;br /&gt;
&lt;br /&gt;
    SSLCertificateFile **********&lt;br /&gt;
    SSLCertificateKeyFile **********&lt;br /&gt;
    SSLCertificateChainFile **********&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Files ~ &amp;quot;\.(cgi|shtml|phtml|php3?)$&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Files&amp;gt;&lt;br /&gt;
    &amp;lt;Directory &amp;quot;/var/www/www/cgi-bin&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Directory&amp;gt;&lt;br /&gt;
&lt;br /&gt;
    SetEnvIf User-Agent &amp;quot;.*MSIE.*&amp;quot; \&lt;br /&gt;
         nokeepalive ssl-unclean-shutdown \&lt;br /&gt;
         downgrade-1.0 force-response-1.0&lt;br /&gt;
&lt;br /&gt;
    CustomLog /var/log/httpd/ssl_request_log   ssl_combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html index.htm index.php&lt;br /&gt;
    Options +indexes&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I have decided to at least organize by processor architecture (a cluster must be 100% x86_64 or ppc64le, no mixing). The basic data that must be present in the web directory is as follows,&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
[root@pliny web_openshift]# ls /home/web_openshift/&lt;br /&gt;
openshift_ppc64le&lt;br /&gt;
[root@pliny web_openshift]# ls -la /home/web_openshift/openshift_ppc64le/&lt;br /&gt;
total 1787720&lt;br /&gt;
drwxr-xr-x. 3 root root      4096 Jul 27 19:48 .&lt;br /&gt;
drwxr-xr-x. 3 root root        31 Jul 27 19:07 ..&lt;br /&gt;
-rw-r--r--. 1 root root    288355 Jul 27 19:06 bootstrap.ign&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 master.ign&lt;br /&gt;
drwxr-xr-x. 2 root root        63 Jul 27 19:06 old&lt;br /&gt;
-rw-r--r--. 1 root root  80882648 Dec 15  2020 rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root  26903229 Dec 15  2020 rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
-rw-r--r--. 1 root root 918583296 Dec 15  2020 rhcos-4.6.8-ppc64le-live.ppc64le.iso&lt;br /&gt;
-rw-r--r--. 1 root root 803940864 Dec 15  2020 rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root       354 Jul 27 19:08 .treeinfo&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 worker.ign&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Note that the .ign files generated by ''openshift_install'' are, by default, chmod 600. If these are copied to the web server without resetting to 644 (go+r), it will fail to serve them up and the installer will explode on the launchpad without generating useful error output.''' My prepare_install.sh script performs this change automatically if used.&lt;br /&gt;
&lt;br /&gt;
Certain information must be present in the .treeinfo file:&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@pliny openshift_ppc64le]# cat .treeinfo &lt;br /&gt;
[general]&lt;br /&gt;
name = CentOS-7&lt;br /&gt;
family = CentOS&lt;br /&gt;
timestamp = 1587405659.3&lt;br /&gt;
variant =&lt;br /&gt;
version = 7&lt;br /&gt;
packagedir =&lt;br /&gt;
arch = ppc64le&lt;br /&gt;
&lt;br /&gt;
[stage2]&lt;br /&gt;
mainimage = rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
&lt;br /&gt;
[images-ppc64le]&lt;br /&gt;
kernel = rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
initrd = rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
boot.iso = rhcos-4.6.8-ppc64le-live.ppc64le.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Of course, the version given (4.6.8 here) needs to match the files actually present.&lt;br /&gt;
&lt;br /&gt;
== DNS setup ==&lt;br /&gt;
&lt;br /&gt;
Our DNS server identifies several private TLDs, including one (.stor) for the private ethernet interfaces of our nodes, for OACISS-local IP addresses. As our openshift setup will not be publicly reachable, we make it live entirely on this private TLD.&lt;br /&gt;
&lt;br /&gt;
First, we create for it the ''openshift.stor'' domain within the DNS private view,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;zone &amp;quot;openshift.stor&amp;quot; in {&lt;br /&gt;
        type master;&lt;br /&gt;
        masterfile-format text;&lt;br /&gt;
        file &amp;quot;openshift_forward.zone&amp;quot;;&lt;br /&gt;
        allow-update { none; };&lt;br /&gt;
        allow-transfer { private_servers; };&lt;br /&gt;
        allow-query { trusted_nets; };&lt;br /&gt;
};&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As we can only define one reverse lookup table for the 172.17.0.0/16 network, the reverse records must go under the storage.reverse file.&lt;br /&gt;
&lt;br /&gt;
The openshift_forward.zone file:&lt;br /&gt;
&amp;lt;pre&amp;gt;$TTL 2h&lt;br /&gt;
@                       IN      SOA     ns.nic.local. systems.nic.uoregon.edu. (&lt;br /&gt;
                                        2021072706      ; Serial number&lt;br /&gt;
                                        21600           ; Refresh(6hrs)&lt;br /&gt;
                                        1800            ; Retry(30min)&lt;br /&gt;
                                        1209600         ; Expire(2wks)&lt;br /&gt;
                                        432000 )        ; Minimum(5dys)&lt;br /&gt;
; vim: ts=4:&lt;br /&gt;
; Name servers.&lt;br /&gt;
&lt;br /&gt;
                        IN      NS      fripp.nic.local.&lt;br /&gt;
&lt;br /&gt;
; Openshift virtual machines&lt;br /&gt;
client                  IN      A       172.17.100.100&lt;br /&gt;
api                     IN      A       172.17.100.100&lt;br /&gt;
api-int                 IN      A       172.17.100.100&lt;br /&gt;
*.apps                  IN      A       172.17.100.100&lt;br /&gt;
&lt;br /&gt;
bootstrap               IN      A       172.17.100.110&lt;br /&gt;
&lt;br /&gt;
master1                 IN      A       172.17.100.101&lt;br /&gt;
master2                 IN      A       172.17.100.102&lt;br /&gt;
master3                 IN      A       172.17.100.103&lt;br /&gt;
worker1                 IN      A       172.17.100.104&lt;br /&gt;
worker2                 IN      A       172.17.100.105&lt;br /&gt;
worker3                 IN      A       172.17.100.106&lt;br /&gt;
worker4                 IN      A       172.17.100.107&lt;br /&gt;
worker5                 IN      A       172.17.100.108&lt;br /&gt;
worker6                 IN      A       172.17.100.109&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
And the relevant entries in storage_reverse.zone:&lt;br /&gt;
&amp;lt;pre&amp;gt;; OpenSHIFT virtual machine reverse lookups&lt;br /&gt;
101.100         IN      PTR     master1.openshift.stor.&lt;br /&gt;
102.100         IN      PTR     master2.openshift.stor.&lt;br /&gt;
103.100         IN      PTR     master3.openshift.stor.&lt;br /&gt;
&lt;br /&gt;
104.100         IN      PTR     worker1.openshift.stor.&lt;br /&gt;
105.100         IN      PTR     worker2.openshift.stor.&lt;br /&gt;
106.100         IN      PTR     worker3.openshift.stor.&lt;br /&gt;
107.100         IN      PTR     worker4.openshift.stor.&lt;br /&gt;
108.100         IN      PTR     worker5.openshift.stor.&lt;br /&gt;
109.100         IN      PTR     worker6.openshift.stor.&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, as is easily forgotten, that the reverse lookups must terminate with .stor. and not just .stor or reverse resolution does not work the way you expect it to :)&lt;br /&gt;
&lt;br /&gt;
The above establishes hostnames for three masters in a quorum and provisions hostnames for up to six workers.&lt;br /&gt;
&lt;br /&gt;
Do not forget to increment the dns serial numbers when this is edited!&lt;br /&gt;
&lt;br /&gt;
== Client VM ==&lt;br /&gt;
&lt;br /&gt;
As the initial entry in the OpenShift saga, we create a virtual machine named (cunningly) ''client'' from a pulled down Centos 8.3 live dvd image,&lt;br /&gt;
&amp;lt;pre&amp;gt;virt-install --virt-type=kvm --name client --memory 2048 --vcpus=2 --os-variant=rhel8.3 --cdrom=/var/lib/libvirt/boot/CentOS-8.3.2011-ppc64le-dvd1.iso --network=network=ocp,model=virtio --console=pty,target_type=virtio --disk path=/var/lib/libvirt/images/centos8.qcow2,size=20,bus=virtio,format=qcow2 --serial pty --graphics none&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will be the access point for the cluster as well as (in our case) running the load balancer.&lt;br /&gt;
&lt;br /&gt;
Before going any further, best to setup the ssh key as this will be needed shortly:&lt;br /&gt;
&amp;lt;pre&amp;gt;ssh-keygen  -t ed25519&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Load balancer ===&lt;br /&gt;
&lt;br /&gt;
OpenShift requires some kind of load balancer to provide a central access point that mediates API access to the worker nodes. We install the HAProxy load balancer onto the client VM.&lt;br /&gt;
&lt;br /&gt;
In higher performance situations, this balancer would be its own (more powerful) VM, its own machine, or in a large-scale datacenter a piece of expensive hardware.&lt;br /&gt;
&lt;br /&gt;
Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The /etc/haproxy/haproxy.cfg file we use:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#---------------------------------------------------------------------&lt;br /&gt;
# Example configuration for a possible web application.  See the&lt;br /&gt;
# full configuration options online.&lt;br /&gt;
#&lt;br /&gt;
#   https://www.haproxy.org/download/1.8/doc/configuration.txt&lt;br /&gt;
#&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# Global settings&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
global&lt;br /&gt;
    # to have these messages end up in /var/log/haproxy.log you will&lt;br /&gt;
    # need to:&lt;br /&gt;
    #&lt;br /&gt;
    # 1) configure syslog to accept network log events.  This is done&lt;br /&gt;
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in&lt;br /&gt;
    #    /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    # 2) configure local2 events to go to the /var/log/haproxy.log&lt;br /&gt;
    #   file. A line like the following can be added to&lt;br /&gt;
    #   /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    #    local2.*                       /var/log/haproxy.log&lt;br /&gt;
    #&lt;br /&gt;
    log         127.0.0.1 local2&lt;br /&gt;
&lt;br /&gt;
    chroot      /var/lib/haproxy&lt;br /&gt;
    pidfile     /var/run/haproxy.pid&lt;br /&gt;
    maxconn     4000&lt;br /&gt;
    user        haproxy&lt;br /&gt;
    group       haproxy&lt;br /&gt;
    daemon&lt;br /&gt;
&lt;br /&gt;
    # turn on stats unix socket&lt;br /&gt;
    stats socket /var/lib/haproxy/stats&lt;br /&gt;
&lt;br /&gt;
    # utilize system-wide crypto-policies&lt;br /&gt;
    ssl-default-bind-ciphers PROFILE=SYSTEM&lt;br /&gt;
    ssl-default-server-ciphers PROFILE=SYSTEM&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# common defaults that all the 'listen' and 'backend' sections will&lt;br /&gt;
# use if not designated in their block&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
defaults&lt;br /&gt;
    log                     global&lt;br /&gt;
    option                  httplog&lt;br /&gt;
    option                  dontlognull&lt;br /&gt;
    option http-server-close&lt;br /&gt;
    option forwardfor       except 127.0.0.0/8&lt;br /&gt;
    option                  redispatch&lt;br /&gt;
    retries                 3&lt;br /&gt;
    timeout http-request    10s&lt;br /&gt;
    timeout queue           1m&lt;br /&gt;
    timeout connect         10s&lt;br /&gt;
    timeout client          30s&lt;br /&gt;
    timeout server          30s&lt;br /&gt;
    timeout http-keep-alive 10s&lt;br /&gt;
    timeout check           10s&lt;br /&gt;
    maxconn                 3000&lt;br /&gt;
&lt;br /&gt;
frontend kubernetes_api&lt;br /&gt;
    bind 172.17.100.100:6443&lt;br /&gt;
    default_backend kubernetes_api&lt;br /&gt;
&lt;br /&gt;
backend kubernetes_api&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:6443 check&lt;br /&gt;
    server master1 master1.openshift.stor:6443 check&lt;br /&gt;
    server master2 master2.openshift.stor:6443 check&lt;br /&gt;
    server master3 master3.openshift.stor:6443 check&lt;br /&gt;
&lt;br /&gt;
frontend machine_config&lt;br /&gt;
    bind 172.17.100.100:22623&lt;br /&gt;
    default_backend machine_config&lt;br /&gt;
&lt;br /&gt;
backend machine_config&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:22623 check&lt;br /&gt;
    server master1 master1.openshift.stor:22623 check&lt;br /&gt;
    server master2 master2.openshift.stor:22623 check&lt;br /&gt;
    server master3 master3.openshift.stor:22623 check&lt;br /&gt;
&lt;br /&gt;
frontend router_https&lt;br /&gt;
    bind 172.17.100.100:443&lt;br /&gt;
    default_backend router_https&lt;br /&gt;
&lt;br /&gt;
backend router_https&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server worker1 worker1.openshift.stor:443 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:443 check&lt;br /&gt;
&lt;br /&gt;
frontend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    option httplog&lt;br /&gt;
    bind 172.17.100.100:80&lt;br /&gt;
    default_backend router_http&lt;br /&gt;
&lt;br /&gt;
backend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    server worker1 worker1.openshift.stor:80 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:80 check&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SElinux and firewall compatibility instructions are mandatory at this juncture:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;semanage  port -a -t http_port_t -p tcp 22623&lt;br /&gt;
semanage  port -a -t http_port_t -p tcp 6443&lt;br /&gt;
firewall-cmd --add-port=6443/tcp&lt;br /&gt;
firewall-cmd --add-port=22623/tcp&lt;br /&gt;
firewall-cmd --runtime-to-permanent&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, this assumes (correctly in the OACISS case) that the client.openshift.stor VM has only a single interface which it default places in the 'public' zone.&lt;br /&gt;
&lt;br /&gt;
= Setup process =&lt;br /&gt;
&lt;br /&gt;
== Ignition files ==&lt;br /&gt;
&lt;br /&gt;
After downloading &amp;lt;pre&amp;gt;openshift-client-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and &amp;lt;pre&amp;gt;openshift-install-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and unpacking them in /root/OCP/ on the client, it is time to generate the Ignition files that will automagically configure the virtual machines.&lt;br /&gt;
&lt;br /&gt;
These are created by openshift-install after reading a .yaml configuration file.&lt;br /&gt;
&lt;br /&gt;
This is the yaml given on the openshift install site as an example of a minimal configuration. Because we are installing the workers manually we must state 0 replicas for workers. Note that openshift-install helpfully deletes the input configuration yaml file, so this backup must be copied each time it is run...&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat install-config.yaml.bak &lt;br /&gt;
apiVersion: v1&lt;br /&gt;
baseDomain: stor&lt;br /&gt;
compute:&lt;br /&gt;
- hyperthreading: Enabled&lt;br /&gt;
  name: worker&lt;br /&gt;
  replicas: 0&lt;br /&gt;
controlPlane:&lt;br /&gt;
  hyperthreading: Enabled&lt;br /&gt;
  name: master&lt;br /&gt;
  replicas: 3&lt;br /&gt;
metadata:&lt;br /&gt;
  name: openshift&lt;br /&gt;
networking:&lt;br /&gt;
  clusterNetwork:&lt;br /&gt;
  - cidr: 10.128.0.0/14&lt;br /&gt;
    hostPrefix: 23&lt;br /&gt;
  networkType: OpenShiftSDN&lt;br /&gt;
  serviceNetwork:&lt;br /&gt;
  - 172.30.0.0/16&lt;br /&gt;
platform:&lt;br /&gt;
  none: {}&lt;br /&gt;
fips: false&lt;br /&gt;
pullSecret: '{}'&lt;br /&gt;
sshKey: 'ssh-ed25519 AAAA************ root@client.openshift.stor'&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Naturally, of course, the actual pullSecret '''and client SSH root trust key should be present.'''&lt;br /&gt;
&lt;br /&gt;
I have packaged the sequence of steps required next into a helpful prepare_install.sh script,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat prepare_install.sh p&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Deleting existing install logs and ign files&amp;quot;&lt;br /&gt;
rm -rf .openshift_install.log .openshift_install_state.json metadata.json bootstrap.ign worker.ign master.ign auth&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;cp install-config.yaml.bak install-config.yaml&amp;quot;&lt;br /&gt;
cp install-config.yaml.bak install-config.yaml&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating manifests&amp;quot;&lt;br /&gt;
./openshift-install create manifests --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating Ignition config files&amp;quot;&lt;br /&gt;
./openshift-install create ignition-configs --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Copying to web server on Pliny&amp;quot;&lt;br /&gt;
chmod go+r *ign&lt;br /&gt;
scp *ign root@pliny:/home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
cp -f /root/OCP/auth/kubeconfig /root/.kube/config&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Install bootstrap node ==&lt;br /&gt;
&lt;br /&gt;
Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;vina# virt-install \&lt;br /&gt;
--name bootstrap \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/bootstrap.qcow2,size=20,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12:80/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It will take something like 4 minutes for this to run and crank the bootstrap machine.&lt;br /&gt;
&lt;br /&gt;
Once this is done, ssh from client to core@bootstrap.openshift.stor and run 'journalctl | grep -i expired', and hopefully no output appears.&lt;br /&gt;
&lt;br /&gt;
If this succeeds, proceed.&lt;br /&gt;
&lt;br /&gt;
== Install master nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the bootstrap node is online, we can initiate installation of the master nodes,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
&lt;br /&gt;
virt-install \&lt;br /&gt;
--name master$NODE \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/master$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The $NODE substitution increments the virtual disk name, IP and hostname appropriately. We, here, have just the three master nodes as 100.101, 2 and 3.&lt;br /&gt;
&lt;br /&gt;
These should take 5-ish minutes to install, boot, self-update and reboot before they try and contact the hive mind.&lt;br /&gt;
&lt;br /&gt;
Several problems can manifest at this point, all caused by misconfiguration on the load balancer,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   88.232297] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&lt;br /&gt;
[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The problem lies in the haproxy configuration file. Under 'global' do not have 'mode http'. This is given correctly in the haproxy config above.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (31min 30s / no limit)[ 1893.903489] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #381&lt;br /&gt;
[ 1893.921039] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: x509: certificate signed by unknown authority&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This occurred when, in initially configuring haproxy, I accidentally told the forwarder for port 22623 to talk to master*:6443.&lt;br /&gt;
&lt;br /&gt;
Another possible problem is&lt;br /&gt;
&amp;lt;pre&amp;gt;[***   ] A start job is running for Ignition (fetch) (1min 45s / no limit)[  108.236817] ignition[690]: GET https://api-int.openshift.stor:22623/config/master: attempt #25&lt;br /&gt;
[  108.249306] ignition[690]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: dial tcp 172.17.100.100:22623: connect: no route to host&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This error is provoked for several possible reasons relating to the load balancer setup (See load balancer section). Most likely either&lt;br /&gt;
* Balancer not running [TCP/SYN rejected]&lt;br /&gt;
* Firewall misconfigured [TCP/SYN being dropped]&lt;br /&gt;
&lt;br /&gt;
== Install completion and bootstrap shutdown ==&lt;br /&gt;
&lt;br /&gt;
From the OCP directory on the client, once the master nodes launch into their self-setup process, run&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;./openshift-install --dir=./ wait-for bootstrap-complete --log-level=info&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This command will block your terminal window until the master nodes finish setting themselves up (a 10+ minute endeavour in my case), thereupon informing you it is safe to shut the bootstrap machine down; &amp;lt;pre&amp;gt;virsh shutdown bootstrap&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, running &amp;lt;pre&amp;gt;[root@client OCP]# ./oc get co&amp;lt;/pre&amp;gt; should vomit out 30 or so lines of the form &amp;quot;NAME [same version] True ...&amp;quot;.&lt;br /&gt;
&lt;br /&gt;
If we get nodes, we now (technically) have a working cluster,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Worker install ==&lt;br /&gt;
&lt;br /&gt;
Now it is time to install the worker nodes. This may be done on the same machine, or across whatever real machines are going to run the cluster,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
virt-install \&lt;br /&gt;
--name worker$NODE \&lt;br /&gt;
--vcpus 32 \&lt;br /&gt;
--ram 65536 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/worker$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Like the install commands for master nodes, the commands for the workers are clones, save for incrementing the IP addresses, workerN hostnames and the virtual disk names.&lt;br /&gt;
&lt;br /&gt;
== Worker install on IBM s924 ==&lt;br /&gt;
&lt;br /&gt;
First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).&lt;br /&gt;
&lt;br /&gt;
Then we must ssh to the VIOS and&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup_env&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
scp erik-k@172.17.202.18:~/downloads/rhcos-4.6.8-ppc64le-live.ppc64le.iso ./&amp;lt;/pre&amp;gt;&lt;br /&gt;
to copy the rhcos installer image to the vios, then use the system -&amp;gt; virtual storage -&amp;gt; vios -&amp;gt; manage -&amp;gt; virtual optical media to add it and assign to the rhcos LPAR.&lt;br /&gt;
&lt;br /&gt;
Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS live-installer grub screen.&lt;br /&gt;
&lt;br /&gt;
Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to a normal terminal to substitute:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Then agonizingly 10-finger this into the vHMC terminal. This should run through a rapid Linux install, grab the rhcos image from Pliny (172.17.202.12), drop it onto the boot disk, then immediately restart.&lt;br /&gt;
&lt;br /&gt;
Now interrupt the boot, shut the partition off, and remove the virtual optical disk so we do not pop back into the rhcos live boot. It will likely be necessary to interrupt the bootp and specify the attached vSCSI hard disk as the boot device.&lt;br /&gt;
&lt;br /&gt;
The partition SMS will now go through the bootp try/fail sequence, then drop into the bootloader that the loader on the virtual disk and initiate the &amp;quot;real&amp;quot; install.&lt;br /&gt;
&lt;br /&gt;
Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.&lt;br /&gt;
&lt;br /&gt;
== Approve new nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the consoles for the worker nodes are sitting at the login prompt, we can add them to the cluster.&lt;br /&gt;
&lt;br /&gt;
Running &amp;lt;pre&amp;gt;./oc get csr&amp;lt;/pre&amp;gt; will show that we have two key requests waiting from the workers.&lt;br /&gt;
&lt;br /&gt;
Do &amp;lt;pre&amp;gt;./oc adm certificate approve $NAME&amp;lt;/pre&amp;gt; for each of the two NAMEd requests to inject the nanoprobes and make them part of the collective.&lt;br /&gt;
&lt;br /&gt;
After about 10-15 seconds we can get nodes again and see they have appeared,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker          109s   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker          102s   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
We note that the master nodes are schedulable because we originally set ourselves up with no worker nodes. Running&lt;br /&gt;
&amp;lt;pre&amp;gt;./oc edit schedulers.config.openshift.io cluster&amp;lt;/pre&amp;gt;&lt;br /&gt;
And edit the line near the bottom for master schedulable from 'true' to 'false'. Now we have what we want:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES    AGE   VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker   10m   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker   10m   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Post-install smart moves ==&lt;br /&gt;
&lt;br /&gt;
Now that it's working, another very minor not at all noteworthy thing to mention...&lt;br /&gt;
&lt;br /&gt;
If the system ever goes down for more than 24 hr, it will be impossible to restart. So, now that it's working, this would be a great time to go run 'virsh list' and then 'virsh autostart X' all the domains: client, master[1 | 2 | 3] and worker[1 | 2] to make sure they come back up even if the host restarts.&lt;br /&gt;
&lt;br /&gt;
Now sit down and pour yourself a nice scotch, you deserve it.&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3326</id>
		<title>Procedure:OpenshiftInstall</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3326"/>
		<updated>2022-01-05T03:18:38Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Worker install on IBM s924 */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page documents the long, painful and complex process of installing OpenShift in a manner that will hopefully reveal the numerous rakes in the grass and avoid a repeat of the famous Sideshow Bob scene.&lt;br /&gt;
&lt;br /&gt;
The big picture:&lt;br /&gt;
&lt;br /&gt;
asdfasdfasdf&lt;br /&gt;
&lt;br /&gt;
= Outside guidance URLs =&lt;br /&gt;
&lt;br /&gt;
* The ostensible guide: https://docs.openshift.com/container-platform/4.6/installing/installing_bare_metal/installing-bare-metal.html#installation-obtaining-installer_installing-bare-metal&lt;br /&gt;
** This reads a lot like a semi-organized stream of consciousness, simply proceeding from A to B to C with no indexing and few explanations.&lt;br /&gt;
* https://cloud.redhat.com/openshift/downloads&lt;br /&gt;
** This leads to the openshift mirror repos where the openshift_client and openshift_installer tar files are found&lt;br /&gt;
* https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos&lt;br /&gt;
** '''PLEASE NOTE''': It is absolutely ''CRITICAL'' to have the SAME versions of the client, installer and rhcos&lt;br /&gt;
** Navigate specifically to version/xx, do not just grab 'stable'.&lt;br /&gt;
* https://www.redhat.com/en/blog/installing-openshift-41-using-libvirt-and-kvm&lt;br /&gt;
** This blog page provides some useful guidance and especially a useful template for the HAProxy configuration&lt;br /&gt;
&lt;br /&gt;
I note that this document describes the installation of a specific and isolated OC cluster.&lt;br /&gt;
&lt;br /&gt;
The OpenShift client system is itself a small VM named client.openshift.stor; This client also runs the proxy/load balancer.&lt;br /&gt;
&lt;br /&gt;
= External prerequisites =&lt;br /&gt;
&lt;br /&gt;
Openshift requires several services/components external to itself in order to work. These are,&lt;br /&gt;
* Web server: for serving up open RHCOS images and Ignition files to the installer&lt;br /&gt;
* Ethernet virbr: As this instance considers the installation of the entire cluster onto VMs living on a single real host, a virtual ethernet bridge must be established to facilitate communication&lt;br /&gt;
* Ethernet NAT masquerade: The cluster lives on a private LAN, and in this case the host machine is configured to provide NAT service to the OC virtual machines&lt;br /&gt;
* DNS: Openshift requires for itself a subdomain and a certain set of forward- and reverse-defined DNS entries.&lt;br /&gt;
* Load balancer: In this case, we install HAProxy on the client system and utilize it as the balancer. Alternatively, a dedicated balancer (VM, or machine, or hardware appliance) may be used.&lt;br /&gt;
&lt;br /&gt;
== Topology Foreword ==&lt;br /&gt;
&lt;br /&gt;
For clarity: The topology of connectivity utilized is&lt;br /&gt;
&lt;br /&gt;
(insert image here)&lt;br /&gt;
&lt;br /&gt;
== Ethernet connectivity ==&lt;br /&gt;
&lt;br /&gt;
The ethernet setup required takes two steps. First, a virtual ethernet bridge (software Level 2) must be setup on each VM host's private ethernet interface. This will create a br0 interface for the virtual bridge. The hardware en___ interface will lose its IP and be slaved to the bridge, and the br0 interface will acquire the IP.&lt;br /&gt;
&lt;br /&gt;
Now, qemu-kvm virtual machines will be able to connect to the br0 bridge, which (from their perspective) is as good as being on the physical Ethernet switch.&lt;br /&gt;
&lt;br /&gt;
As the second step, an IP masquerade/NAT is established for the cluster IPs such that they are able to name the host IP as their gateway and reach the wider Internet (to download updates and containers).&lt;br /&gt;
&lt;br /&gt;
=== Bridge setup ===&lt;br /&gt;
&lt;br /&gt;
Creating the virbr is simple on an Ubuntu host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;root@vina:~# cat /etc/netplan/00-installer-config.yaml &lt;br /&gt;
# This is the network config written by 'subiquity'&lt;br /&gt;
network:&lt;br /&gt;
  ethernets:&lt;br /&gt;
    enp1s0f0:&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
    enp1s0f1:&lt;br /&gt;
#      dhcp4: true&lt;br /&gt;
      mtu: 9000&lt;br /&gt;
  version: 2&lt;br /&gt;
  bridges:&lt;br /&gt;
    br0:&lt;br /&gt;
      interfaces: [enp1s0f1]&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
      mtu: 9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that here, mtu=9000 is set because our storage/private ethernet is setup to use jumbo frames and it is quite necessary that the IP MTUs match, or magic packet loss is likely to occur; Software bridges may automatically fragment jumbo frames, but hardware will simply discard overlength packets.&lt;br /&gt;
&lt;br /&gt;
The bridging setup on RHEL can be achieved several ways. I prefer to just punch it straight into /etc/sysconfig/network-scripts/*, as in this example that sets up one of the virbrs on our infrastructure vm node,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-eno1&lt;br /&gt;
TYPE=Ethernet&lt;br /&gt;
NAME=eno1-bridge-slave&lt;br /&gt;
UUID=9e7e3a89-3358-4830-a033-0a4154c68c55&lt;br /&gt;
DEVICE=eno1&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
BRIDGE=br0&lt;br /&gt;
HWADDR=3c:ec:ef:1a:71:3e&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-br0&lt;br /&gt;
STP=no&lt;br /&gt;
BRIDGING_OPTS=priority=32768&lt;br /&gt;
TYPE=Bridge&lt;br /&gt;
PROXY_METHOD=none&lt;br /&gt;
BROWSER_ONLY=no&lt;br /&gt;
BOOTPROTO=dhcp&lt;br /&gt;
DEFROUTE=yes&lt;br /&gt;
IPV4_FAILURE_FATAL=no&lt;br /&gt;
IPV6INIT=yes&lt;br /&gt;
IPV6_AUTOCONF=yes&lt;br /&gt;
IPV6_DEFROUTE=yes&lt;br /&gt;
IPV6_FAILURE_FATAL=no&lt;br /&gt;
IPV6_ADDR_GEN_MODE=stable-privacy&lt;br /&gt;
NAME=br0&lt;br /&gt;
UUID=3549a392-12ac-4c7f-bdd5-4e86d8654ddf&lt;br /&gt;
DEVICE=br0&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
MTU=9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It may be possible to leave STP enabled, however this was found to cause problems in some settings (the interface claims to be activating forever and never enters forwarding - i.e. the host physical ethernet port did not talk STP back)&lt;br /&gt;
&lt;br /&gt;
Once the bridge has been defined, we must inform kvm about it by defining an xml, importing it and marking it to autostart in kvm.&lt;br /&gt;
&lt;br /&gt;
=== IP masquerading ===&lt;br /&gt;
&lt;br /&gt;
Assuming that routing is enabled on the host and normal routing table entries are setup, the following three lines will setup IP masquerade between the public interface ('ifpublic') and br0,&lt;br /&gt;
&amp;lt;pre&amp;gt;iptables -A FORWARD -i br0 -o ifpublic -j ACCEPT&lt;br /&gt;
iptables -A FORWARD -i ifpublic -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT&lt;br /&gt;
iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.&lt;br /&gt;
&lt;br /&gt;
== Web server ==&lt;br /&gt;
&lt;br /&gt;
Some web space must be provisioned that can serve up&lt;br /&gt;
&lt;br /&gt;
* RHCOS images&lt;br /&gt;
* Ignition files&lt;br /&gt;
&lt;br /&gt;
The system at this stage is not paranoid; In our case, I have setup a 10-openshift.conf Apache server bound to Pliny's private network interface (172.17.202.12). Stripping all commentary from the configuration file,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:80&amp;gt;&lt;br /&gt;
    ServerAdmin systems@nic.uoregon.edu&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    DocumentRoot /home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
    # if not specified, the global error log is used&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/error_log&lt;br /&gt;
    CustomLog /var/log/httpd/openshift/access_log combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        options +indexes&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:443&amp;gt;&lt;br /&gt;
    #  General setup for the virtual host&lt;br /&gt;
    DocumentRoot &amp;quot;/home/web_openshift/&amp;quot;&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/ssl_error_log&lt;br /&gt;
    TransferLog /var/log/httpd/openshift/ssl_access_log&lt;br /&gt;
&lt;br /&gt;
    SSLEngine on&lt;br /&gt;
&lt;br /&gt;
    #   SSL Protocol Support:&lt;br /&gt;
    SSLProtocol All -SSLv2 -SSLv3&lt;br /&gt;
    SSLCipherSuite    ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA&lt;br /&gt;
    SSLHonorCipherOrder     on&lt;br /&gt;
&lt;br /&gt;
    SSLCertificateFile **********&lt;br /&gt;
    SSLCertificateKeyFile **********&lt;br /&gt;
    SSLCertificateChainFile **********&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Files ~ &amp;quot;\.(cgi|shtml|phtml|php3?)$&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Files&amp;gt;&lt;br /&gt;
    &amp;lt;Directory &amp;quot;/var/www/www/cgi-bin&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Directory&amp;gt;&lt;br /&gt;
&lt;br /&gt;
    SetEnvIf User-Agent &amp;quot;.*MSIE.*&amp;quot; \&lt;br /&gt;
         nokeepalive ssl-unclean-shutdown \&lt;br /&gt;
         downgrade-1.0 force-response-1.0&lt;br /&gt;
&lt;br /&gt;
    CustomLog /var/log/httpd/ssl_request_log   ssl_combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html index.htm index.php&lt;br /&gt;
    Options +indexes&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I have decided to at least organize by processor architecture (a cluster must be 100% x86_64 or ppc64le, no mixing). The basic data that must be present in the web directory is as follows,&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
[root@pliny web_openshift]# ls /home/web_openshift/&lt;br /&gt;
openshift_ppc64le&lt;br /&gt;
[root@pliny web_openshift]# ls -la /home/web_openshift/openshift_ppc64le/&lt;br /&gt;
total 1787720&lt;br /&gt;
drwxr-xr-x. 3 root root      4096 Jul 27 19:48 .&lt;br /&gt;
drwxr-xr-x. 3 root root        31 Jul 27 19:07 ..&lt;br /&gt;
-rw-r--r--. 1 root root    288355 Jul 27 19:06 bootstrap.ign&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 master.ign&lt;br /&gt;
drwxr-xr-x. 2 root root        63 Jul 27 19:06 old&lt;br /&gt;
-rw-r--r--. 1 root root  80882648 Dec 15  2020 rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root  26903229 Dec 15  2020 rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
-rw-r--r--. 1 root root 918583296 Dec 15  2020 rhcos-4.6.8-ppc64le-live.ppc64le.iso&lt;br /&gt;
-rw-r--r--. 1 root root 803940864 Dec 15  2020 rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root       354 Jul 27 19:08 .treeinfo&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 worker.ign&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Note that the .ign files generated by ''openshift_install'' are, by default, chmod 600. If these are copied to the web server without resetting to 644 (go+r), it will fail to serve them up and the installer will explode on the launchpad without generating useful error output.''' My prepare_install.sh script performs this change automatically if used.&lt;br /&gt;
&lt;br /&gt;
Certain information must be present in the .treeinfo file:&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@pliny openshift_ppc64le]# cat .treeinfo &lt;br /&gt;
[general]&lt;br /&gt;
name = CentOS-7&lt;br /&gt;
family = CentOS&lt;br /&gt;
timestamp = 1587405659.3&lt;br /&gt;
variant =&lt;br /&gt;
version = 7&lt;br /&gt;
packagedir =&lt;br /&gt;
arch = ppc64le&lt;br /&gt;
&lt;br /&gt;
[stage2]&lt;br /&gt;
mainimage = rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
&lt;br /&gt;
[images-ppc64le]&lt;br /&gt;
kernel = rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
initrd = rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
boot.iso = rhcos-4.6.8-ppc64le-live.ppc64le.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Of course, the version given (4.6.8 here) needs to match the files actually present.&lt;br /&gt;
&lt;br /&gt;
== DNS setup ==&lt;br /&gt;
&lt;br /&gt;
Our DNS server identifies several private TLDs, including one (.stor) for the private ethernet interfaces of our nodes, for OACISS-local IP addresses. As our openshift setup will not be publicly reachable, we make it live entirely on this private TLD.&lt;br /&gt;
&lt;br /&gt;
First, we create for it the ''openshift.stor'' domain within the DNS private view,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;zone &amp;quot;openshift.stor&amp;quot; in {&lt;br /&gt;
        type master;&lt;br /&gt;
        masterfile-format text;&lt;br /&gt;
        file &amp;quot;openshift_forward.zone&amp;quot;;&lt;br /&gt;
        allow-update { none; };&lt;br /&gt;
        allow-transfer { private_servers; };&lt;br /&gt;
        allow-query { trusted_nets; };&lt;br /&gt;
};&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As we can only define one reverse lookup table for the 172.17.0.0/16 network, the reverse records must go under the storage.reverse file.&lt;br /&gt;
&lt;br /&gt;
The openshift_forward.zone file:&lt;br /&gt;
&amp;lt;pre&amp;gt;$TTL 2h&lt;br /&gt;
@                       IN      SOA     ns.nic.local. systems.nic.uoregon.edu. (&lt;br /&gt;
                                        2021072706      ; Serial number&lt;br /&gt;
                                        21600           ; Refresh(6hrs)&lt;br /&gt;
                                        1800            ; Retry(30min)&lt;br /&gt;
                                        1209600         ; Expire(2wks)&lt;br /&gt;
                                        432000 )        ; Minimum(5dys)&lt;br /&gt;
; vim: ts=4:&lt;br /&gt;
; Name servers.&lt;br /&gt;
&lt;br /&gt;
                        IN      NS      fripp.nic.local.&lt;br /&gt;
&lt;br /&gt;
; Openshift virtual machines&lt;br /&gt;
client                  IN      A       172.17.100.100&lt;br /&gt;
api                     IN      A       172.17.100.100&lt;br /&gt;
api-int                 IN      A       172.17.100.100&lt;br /&gt;
*.apps                  IN      A       172.17.100.100&lt;br /&gt;
&lt;br /&gt;
bootstrap               IN      A       172.17.100.110&lt;br /&gt;
&lt;br /&gt;
master1                 IN      A       172.17.100.101&lt;br /&gt;
master2                 IN      A       172.17.100.102&lt;br /&gt;
master3                 IN      A       172.17.100.103&lt;br /&gt;
worker1                 IN      A       172.17.100.104&lt;br /&gt;
worker2                 IN      A       172.17.100.105&lt;br /&gt;
worker3                 IN      A       172.17.100.106&lt;br /&gt;
worker4                 IN      A       172.17.100.107&lt;br /&gt;
worker5                 IN      A       172.17.100.108&lt;br /&gt;
worker6                 IN      A       172.17.100.109&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
And the relevant entries in storage_reverse.zone:&lt;br /&gt;
&amp;lt;pre&amp;gt;; OpenSHIFT virtual machine reverse lookups&lt;br /&gt;
101.100         IN      PTR     master1.openshift.stor.&lt;br /&gt;
102.100         IN      PTR     master2.openshift.stor.&lt;br /&gt;
103.100         IN      PTR     master3.openshift.stor.&lt;br /&gt;
&lt;br /&gt;
104.100         IN      PTR     worker1.openshift.stor.&lt;br /&gt;
105.100         IN      PTR     worker2.openshift.stor.&lt;br /&gt;
106.100         IN      PTR     worker3.openshift.stor.&lt;br /&gt;
107.100         IN      PTR     worker4.openshift.stor.&lt;br /&gt;
108.100         IN      PTR     worker5.openshift.stor.&lt;br /&gt;
109.100         IN      PTR     worker6.openshift.stor.&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, as is easily forgotten, that the reverse lookups must terminate with .stor. and not just .stor or reverse resolution does not work the way you expect it to :)&lt;br /&gt;
&lt;br /&gt;
The above establishes hostnames for three masters in a quorum and provisions hostnames for up to six workers.&lt;br /&gt;
&lt;br /&gt;
Do not forget to increment the dns serial numbers when this is edited!&lt;br /&gt;
&lt;br /&gt;
== Client VM ==&lt;br /&gt;
&lt;br /&gt;
As the initial entry in the OpenShift saga, we create a virtual machine named (cunningly) ''client'' from a pulled down Centos 8.3 live dvd image,&lt;br /&gt;
&amp;lt;pre&amp;gt;virt-install --virt-type=kvm --name client --memory 2048 --vcpus=2 --os-variant=rhel8.3 --cdrom=/var/lib/libvirt/boot/CentOS-8.3.2011-ppc64le-dvd1.iso --network=network=ocp,model=virtio --console=pty,target_type=virtio --disk path=/var/lib/libvirt/images/centos8.qcow2,size=20,bus=virtio,format=qcow2 --serial pty --graphics none&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will be the access point for the cluster as well as (in our case) running the load balancer.&lt;br /&gt;
&lt;br /&gt;
Before going any further, best to setup the ssh key as this will be needed shortly:&lt;br /&gt;
&amp;lt;pre&amp;gt;ssh-keygen  -t ed25519&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Load balancer ===&lt;br /&gt;
&lt;br /&gt;
OpenShift requires some kind of load balancer to provide a central access point that mediates API access to the worker nodes. We install the HAProxy load balancer onto the client VM.&lt;br /&gt;
&lt;br /&gt;
In higher performance situations, this balancer would be its own (more powerful) VM, its own machine, or in a large-scale datacenter a piece of expensive hardware.&lt;br /&gt;
&lt;br /&gt;
Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The /etc/haproxy/haproxy.cfg file we use:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#---------------------------------------------------------------------&lt;br /&gt;
# Example configuration for a possible web application.  See the&lt;br /&gt;
# full configuration options online.&lt;br /&gt;
#&lt;br /&gt;
#   https://www.haproxy.org/download/1.8/doc/configuration.txt&lt;br /&gt;
#&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# Global settings&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
global&lt;br /&gt;
    # to have these messages end up in /var/log/haproxy.log you will&lt;br /&gt;
    # need to:&lt;br /&gt;
    #&lt;br /&gt;
    # 1) configure syslog to accept network log events.  This is done&lt;br /&gt;
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in&lt;br /&gt;
    #    /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    # 2) configure local2 events to go to the /var/log/haproxy.log&lt;br /&gt;
    #   file. A line like the following can be added to&lt;br /&gt;
    #   /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    #    local2.*                       /var/log/haproxy.log&lt;br /&gt;
    #&lt;br /&gt;
    log         127.0.0.1 local2&lt;br /&gt;
&lt;br /&gt;
    chroot      /var/lib/haproxy&lt;br /&gt;
    pidfile     /var/run/haproxy.pid&lt;br /&gt;
    maxconn     4000&lt;br /&gt;
    user        haproxy&lt;br /&gt;
    group       haproxy&lt;br /&gt;
    daemon&lt;br /&gt;
&lt;br /&gt;
    # turn on stats unix socket&lt;br /&gt;
    stats socket /var/lib/haproxy/stats&lt;br /&gt;
&lt;br /&gt;
    # utilize system-wide crypto-policies&lt;br /&gt;
    ssl-default-bind-ciphers PROFILE=SYSTEM&lt;br /&gt;
    ssl-default-server-ciphers PROFILE=SYSTEM&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# common defaults that all the 'listen' and 'backend' sections will&lt;br /&gt;
# use if not designated in their block&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
defaults&lt;br /&gt;
    log                     global&lt;br /&gt;
    option                  httplog&lt;br /&gt;
    option                  dontlognull&lt;br /&gt;
    option http-server-close&lt;br /&gt;
    option forwardfor       except 127.0.0.0/8&lt;br /&gt;
    option                  redispatch&lt;br /&gt;
    retries                 3&lt;br /&gt;
    timeout http-request    10s&lt;br /&gt;
    timeout queue           1m&lt;br /&gt;
    timeout connect         10s&lt;br /&gt;
    timeout client          30s&lt;br /&gt;
    timeout server          30s&lt;br /&gt;
    timeout http-keep-alive 10s&lt;br /&gt;
    timeout check           10s&lt;br /&gt;
    maxconn                 3000&lt;br /&gt;
&lt;br /&gt;
frontend kubernetes_api&lt;br /&gt;
    bind 172.17.100.100:6443&lt;br /&gt;
    default_backend kubernetes_api&lt;br /&gt;
&lt;br /&gt;
backend kubernetes_api&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:6443 check&lt;br /&gt;
    server master1 master1.openshift.stor:6443 check&lt;br /&gt;
    server master2 master2.openshift.stor:6443 check&lt;br /&gt;
    server master3 master3.openshift.stor:6443 check&lt;br /&gt;
&lt;br /&gt;
frontend machine_config&lt;br /&gt;
    bind 172.17.100.100:22623&lt;br /&gt;
    default_backend machine_config&lt;br /&gt;
&lt;br /&gt;
backend machine_config&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:22623 check&lt;br /&gt;
    server master1 master1.openshift.stor:22623 check&lt;br /&gt;
    server master2 master2.openshift.stor:22623 check&lt;br /&gt;
    server master3 master3.openshift.stor:22623 check&lt;br /&gt;
&lt;br /&gt;
frontend router_https&lt;br /&gt;
    bind 172.17.100.100:443&lt;br /&gt;
    default_backend router_https&lt;br /&gt;
&lt;br /&gt;
backend router_https&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server worker1 worker1.openshift.stor:443 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:443 check&lt;br /&gt;
&lt;br /&gt;
frontend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    option httplog&lt;br /&gt;
    bind 172.17.100.100:80&lt;br /&gt;
    default_backend router_http&lt;br /&gt;
&lt;br /&gt;
backend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    server worker1 worker1.openshift.stor:80 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:80 check&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SElinux and firewall compatibility instructions are mandatory at this juncture:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;semanage  port -a -t http_port_t -p tcp 22623&lt;br /&gt;
semanage  port -a -t http_port_t -p tcp 6443&lt;br /&gt;
firewall-cmd --add-port=6443/tcp&lt;br /&gt;
firewall-cmd --add-port=22623/tcp&lt;br /&gt;
firewall-cmd --runtime-to-permanent&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, this assumes (correctly in the OACISS case) that the client.openshift.stor VM has only a single interface which it default places in the 'public' zone.&lt;br /&gt;
&lt;br /&gt;
= Setup process =&lt;br /&gt;
&lt;br /&gt;
== Ignition files ==&lt;br /&gt;
&lt;br /&gt;
After downloading &amp;lt;pre&amp;gt;openshift-client-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and &amp;lt;pre&amp;gt;openshift-install-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and unpacking them in /root/OCP/ on the client, it is time to generate the Ignition files that will automagically configure the virtual machines.&lt;br /&gt;
&lt;br /&gt;
These are created by openshift-install after reading a .yaml configuration file.&lt;br /&gt;
&lt;br /&gt;
This is the yaml given on the openshift install site as an example of a minimal configuration. Because we are installing the workers manually we must state 0 replicas for workers. Note that openshift-install helpfully deletes the input configuration yaml file, so this backup must be copied each time it is run...&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat install-config.yaml.bak &lt;br /&gt;
apiVersion: v1&lt;br /&gt;
baseDomain: stor&lt;br /&gt;
compute:&lt;br /&gt;
- hyperthreading: Enabled&lt;br /&gt;
  name: worker&lt;br /&gt;
  replicas: 0&lt;br /&gt;
controlPlane:&lt;br /&gt;
  hyperthreading: Enabled&lt;br /&gt;
  name: master&lt;br /&gt;
  replicas: 3&lt;br /&gt;
metadata:&lt;br /&gt;
  name: openshift&lt;br /&gt;
networking:&lt;br /&gt;
  clusterNetwork:&lt;br /&gt;
  - cidr: 10.128.0.0/14&lt;br /&gt;
    hostPrefix: 23&lt;br /&gt;
  networkType: OpenShiftSDN&lt;br /&gt;
  serviceNetwork:&lt;br /&gt;
  - 172.30.0.0/16&lt;br /&gt;
platform:&lt;br /&gt;
  none: {}&lt;br /&gt;
fips: false&lt;br /&gt;
pullSecret: '{}'&lt;br /&gt;
sshKey: 'ssh-ed25519 AAAA************ root@client.openshift.stor'&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Naturally, of course, the actual pullSecret '''and client SSH root trust key should be present.'''&lt;br /&gt;
&lt;br /&gt;
I have packaged the sequence of steps required next into a helpful prepare_install.sh script,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat prepare_install.sh p&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Deleting existing install logs and ign files&amp;quot;&lt;br /&gt;
rm -rf .openshift_install.log .openshift_install_state.json metadata.json bootstrap.ign worker.ign master.ign auth&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;cp install-config.yaml.bak install-config.yaml&amp;quot;&lt;br /&gt;
cp install-config.yaml.bak install-config.yaml&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating manifests&amp;quot;&lt;br /&gt;
./openshift-install create manifests --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating Ignition config files&amp;quot;&lt;br /&gt;
./openshift-install create ignition-configs --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Copying to web server on Pliny&amp;quot;&lt;br /&gt;
chmod go+r *ign&lt;br /&gt;
scp *ign root@pliny:/home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
cp -f /root/OCP/auth/kubeconfig /root/.kube/config&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Install bootstrap node ==&lt;br /&gt;
&lt;br /&gt;
Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;vina# virt-install \&lt;br /&gt;
--name bootstrap \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/bootstrap.qcow2,size=20,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12:80/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It will take something like 4 minutes for this to run and crank the bootstrap machine.&lt;br /&gt;
&lt;br /&gt;
Once this is done, ssh from client to core@bootstrap.openshift.stor and run 'journalctl | grep -i expired', and hopefully no output appears.&lt;br /&gt;
&lt;br /&gt;
If this succeeds, proceed.&lt;br /&gt;
&lt;br /&gt;
== Install master nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the bootstrap node is online, we can initiate installation of the master nodes,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
&lt;br /&gt;
virt-install \&lt;br /&gt;
--name master$NODE \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/master$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The $NODE substitution increments the virtual disk name, IP and hostname appropriately. We, here, have just the three master nodes as 100.101, 2 and 3.&lt;br /&gt;
&lt;br /&gt;
These should take 5-ish minutes to install, boot, self-update and reboot before they try and contact the hive mind.&lt;br /&gt;
&lt;br /&gt;
Several problems can manifest at this point, all caused by misconfiguration on the load balancer,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   88.232297] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&lt;br /&gt;
[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The problem lies in the haproxy configuration file. Under 'global' do not have 'mode http'. This is given correctly in the haproxy config above.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (31min 30s / no limit)[ 1893.903489] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #381&lt;br /&gt;
[ 1893.921039] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: x509: certificate signed by unknown authority&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This occurred when, in initially configuring haproxy, I accidentally told the forwarder for port 22623 to talk to master*:6443.&lt;br /&gt;
&lt;br /&gt;
Another possible problem is&lt;br /&gt;
&amp;lt;pre&amp;gt;[***   ] A start job is running for Ignition (fetch) (1min 45s / no limit)[  108.236817] ignition[690]: GET https://api-int.openshift.stor:22623/config/master: attempt #25&lt;br /&gt;
[  108.249306] ignition[690]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: dial tcp 172.17.100.100:22623: connect: no route to host&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This error is provoked for several possible reasons relating to the load balancer setup (See load balancer section). Most likely either&lt;br /&gt;
* Balancer not running [TCP/SYN rejected]&lt;br /&gt;
* Firewall misconfigured [TCP/SYN being dropped]&lt;br /&gt;
&lt;br /&gt;
== Install completion and bootstrap shutdown ==&lt;br /&gt;
&lt;br /&gt;
From the OCP directory on the client, once the master nodes launch into their self-setup process, run&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;./openshift-install --dir=./ wait-for bootstrap-complete --log-level=info&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This command will block your terminal window until the master nodes finish setting themselves up (a 10+ minute endeavour in my case), thereupon informing you it is safe to shut the bootstrap machine down; &amp;lt;pre&amp;gt;virsh shutdown bootstrap&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, running &amp;lt;pre&amp;gt;[root@client OCP]# ./oc get co&amp;lt;/pre&amp;gt; should vomit out 30 or so lines of the form &amp;quot;NAME [same version] True ...&amp;quot;.&lt;br /&gt;
&lt;br /&gt;
If we get nodes, we now (technically) have a working cluster,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Worker install ==&lt;br /&gt;
&lt;br /&gt;
Now it is time to install the worker nodes. This may be done on the same machine, or across whatever real machines are going to run the cluster,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
virt-install \&lt;br /&gt;
--name worker$NODE \&lt;br /&gt;
--vcpus 32 \&lt;br /&gt;
--ram 65536 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/worker$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Like the install commands for master nodes, the commands for the workers are clones, save for incrementing the IP addresses, workerN hostnames and the virtual disk names.&lt;br /&gt;
&lt;br /&gt;
== Worker install on IBM s924 ==&lt;br /&gt;
&lt;br /&gt;
First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).&lt;br /&gt;
&lt;br /&gt;
Then we must ssh to the VIOS and&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup_env&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
scp erik-k@172.17.202.18:~/downloads/rhcos-4.6.8-ppc64le-live.ppc64le.iso ./&amp;lt;/pre&amp;gt;&lt;br /&gt;
to copy the rhcos installer image to the vios, then use the system -&amp;gt; virtual storage -&amp;gt; vios -&amp;gt; manage -&amp;gt; virtual optical media to add it and assign to the rhcos LPAR.&lt;br /&gt;
&lt;br /&gt;
Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS live-installer grub screen.&lt;br /&gt;
&lt;br /&gt;
Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to a normal terminal to substitute:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Then agonizingly 10-finger this into the vHMC terminal. This should run through a rapid Linux install, grab the rhcos image from Pliny (172.17.202.12), drop it onto the boot disk, then immediately restart.&lt;br /&gt;
&lt;br /&gt;
Now interrupt the boot, shut the partition off, and remove the virtual optical disk so we do not pop back into the rhcos live boot.&lt;br /&gt;
&lt;br /&gt;
The partition SMS will now go through the bootp try/fail sequence, then drop into the bootloader that the loader on the virtual disk and initiate the &amp;quot;real&amp;quot; install.&lt;br /&gt;
&lt;br /&gt;
Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.&lt;br /&gt;
&lt;br /&gt;
== Approve new nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the consoles for the worker nodes are sitting at the login prompt, we can add them to the cluster.&lt;br /&gt;
&lt;br /&gt;
Running &amp;lt;pre&amp;gt;./oc get csr&amp;lt;/pre&amp;gt; will show that we have two key requests waiting from the workers.&lt;br /&gt;
&lt;br /&gt;
Do &amp;lt;pre&amp;gt;./oc adm certificate approve $NAME&amp;lt;/pre&amp;gt; for each of the two NAMEd requests to inject the nanoprobes and make them part of the collective.&lt;br /&gt;
&lt;br /&gt;
After about 10-15 seconds we can get nodes again and see they have appeared,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker          109s   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker          102s   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
We note that the master nodes are schedulable because we originally set ourselves up with no worker nodes. Running&lt;br /&gt;
&amp;lt;pre&amp;gt;./oc edit schedulers.config.openshift.io cluster&amp;lt;/pre&amp;gt;&lt;br /&gt;
And edit the line near the bottom for master schedulable from 'true' to 'false'. Now we have what we want:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES    AGE   VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker   10m   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker   10m   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Post-install smart moves ==&lt;br /&gt;
&lt;br /&gt;
Now that it's working, another very minor not at all noteworthy thing to mention...&lt;br /&gt;
&lt;br /&gt;
If the system ever goes down for more than 24 hr, it will be impossible to restart. So, now that it's working, this would be a great time to go run 'virsh list' and then 'virsh autostart X' all the domains: client, master[1 | 2 | 3] and worker[1 | 2] to make sure they come back up even if the host restarts.&lt;br /&gt;
&lt;br /&gt;
Now sit down and pour yourself a nice scotch, you deserve it.&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3325</id>
		<title>Procedure:OpenshiftInstall</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3325"/>
		<updated>2021-12-23T07:31:03Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Load balancer */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page documents the long, painful and complex process of installing OpenShift in a manner that will hopefully reveal the numerous rakes in the grass and avoid a repeat of the famous Sideshow Bob scene.&lt;br /&gt;
&lt;br /&gt;
The big picture:&lt;br /&gt;
&lt;br /&gt;
asdfasdfasdf&lt;br /&gt;
&lt;br /&gt;
= Outside guidance URLs =&lt;br /&gt;
&lt;br /&gt;
* The ostensible guide: https://docs.openshift.com/container-platform/4.6/installing/installing_bare_metal/installing-bare-metal.html#installation-obtaining-installer_installing-bare-metal&lt;br /&gt;
** This reads a lot like a semi-organized stream of consciousness, simply proceeding from A to B to C with no indexing and few explanations.&lt;br /&gt;
* https://cloud.redhat.com/openshift/downloads&lt;br /&gt;
** This leads to the openshift mirror repos where the openshift_client and openshift_installer tar files are found&lt;br /&gt;
* https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos&lt;br /&gt;
** '''PLEASE NOTE''': It is absolutely ''CRITICAL'' to have the SAME versions of the client, installer and rhcos&lt;br /&gt;
** Navigate specifically to version/xx, do not just grab 'stable'.&lt;br /&gt;
* https://www.redhat.com/en/blog/installing-openshift-41-using-libvirt-and-kvm&lt;br /&gt;
** This blog page provides some useful guidance and especially a useful template for the HAProxy configuration&lt;br /&gt;
&lt;br /&gt;
I note that this document describes the installation of a specific and isolated OC cluster.&lt;br /&gt;
&lt;br /&gt;
The OpenShift client system is itself a small VM named client.openshift.stor; This client also runs the proxy/load balancer.&lt;br /&gt;
&lt;br /&gt;
= External prerequisites =&lt;br /&gt;
&lt;br /&gt;
Openshift requires several services/components external to itself in order to work. These are,&lt;br /&gt;
* Web server: for serving up open RHCOS images and Ignition files to the installer&lt;br /&gt;
* Ethernet virbr: As this instance considers the installation of the entire cluster onto VMs living on a single real host, a virtual ethernet bridge must be established to facilitate communication&lt;br /&gt;
* Ethernet NAT masquerade: The cluster lives on a private LAN, and in this case the host machine is configured to provide NAT service to the OC virtual machines&lt;br /&gt;
* DNS: Openshift requires for itself a subdomain and a certain set of forward- and reverse-defined DNS entries.&lt;br /&gt;
* Load balancer: In this case, we install HAProxy on the client system and utilize it as the balancer. Alternatively, a dedicated balancer (VM, or machine, or hardware appliance) may be used.&lt;br /&gt;
&lt;br /&gt;
== Topology Foreword ==&lt;br /&gt;
&lt;br /&gt;
For clarity: The topology of connectivity utilized is&lt;br /&gt;
&lt;br /&gt;
(insert image here)&lt;br /&gt;
&lt;br /&gt;
== Ethernet connectivity ==&lt;br /&gt;
&lt;br /&gt;
The ethernet setup required takes two steps. First, a virtual ethernet bridge (software Level 2) must be setup on each VM host's private ethernet interface. This will create a br0 interface for the virtual bridge. The hardware en___ interface will lose its IP and be slaved to the bridge, and the br0 interface will acquire the IP.&lt;br /&gt;
&lt;br /&gt;
Now, qemu-kvm virtual machines will be able to connect to the br0 bridge, which (from their perspective) is as good as being on the physical Ethernet switch.&lt;br /&gt;
&lt;br /&gt;
As the second step, an IP masquerade/NAT is established for the cluster IPs such that they are able to name the host IP as their gateway and reach the wider Internet (to download updates and containers).&lt;br /&gt;
&lt;br /&gt;
=== Bridge setup ===&lt;br /&gt;
&lt;br /&gt;
Creating the virbr is simple on an Ubuntu host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;root@vina:~# cat /etc/netplan/00-installer-config.yaml &lt;br /&gt;
# This is the network config written by 'subiquity'&lt;br /&gt;
network:&lt;br /&gt;
  ethernets:&lt;br /&gt;
    enp1s0f0:&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
    enp1s0f1:&lt;br /&gt;
#      dhcp4: true&lt;br /&gt;
      mtu: 9000&lt;br /&gt;
  version: 2&lt;br /&gt;
  bridges:&lt;br /&gt;
    br0:&lt;br /&gt;
      interfaces: [enp1s0f1]&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
      mtu: 9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that here, mtu=9000 is set because our storage/private ethernet is setup to use jumbo frames and it is quite necessary that the IP MTUs match, or magic packet loss is likely to occur; Software bridges may automatically fragment jumbo frames, but hardware will simply discard overlength packets.&lt;br /&gt;
&lt;br /&gt;
The bridging setup on RHEL can be achieved several ways. I prefer to just punch it straight into /etc/sysconfig/network-scripts/*, as in this example that sets up one of the virbrs on our infrastructure vm node,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-eno1&lt;br /&gt;
TYPE=Ethernet&lt;br /&gt;
NAME=eno1-bridge-slave&lt;br /&gt;
UUID=9e7e3a89-3358-4830-a033-0a4154c68c55&lt;br /&gt;
DEVICE=eno1&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
BRIDGE=br0&lt;br /&gt;
HWADDR=3c:ec:ef:1a:71:3e&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-br0&lt;br /&gt;
STP=no&lt;br /&gt;
BRIDGING_OPTS=priority=32768&lt;br /&gt;
TYPE=Bridge&lt;br /&gt;
PROXY_METHOD=none&lt;br /&gt;
BROWSER_ONLY=no&lt;br /&gt;
BOOTPROTO=dhcp&lt;br /&gt;
DEFROUTE=yes&lt;br /&gt;
IPV4_FAILURE_FATAL=no&lt;br /&gt;
IPV6INIT=yes&lt;br /&gt;
IPV6_AUTOCONF=yes&lt;br /&gt;
IPV6_DEFROUTE=yes&lt;br /&gt;
IPV6_FAILURE_FATAL=no&lt;br /&gt;
IPV6_ADDR_GEN_MODE=stable-privacy&lt;br /&gt;
NAME=br0&lt;br /&gt;
UUID=3549a392-12ac-4c7f-bdd5-4e86d8654ddf&lt;br /&gt;
DEVICE=br0&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
MTU=9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It may be possible to leave STP enabled, however this was found to cause problems in some settings (the interface claims to be activating forever and never enters forwarding - i.e. the host physical ethernet port did not talk STP back)&lt;br /&gt;
&lt;br /&gt;
Once the bridge has been defined, we must inform kvm about it by defining an xml, importing it and marking it to autostart in kvm.&lt;br /&gt;
&lt;br /&gt;
=== IP masquerading ===&lt;br /&gt;
&lt;br /&gt;
Assuming that routing is enabled on the host and normal routing table entries are setup, the following three lines will setup IP masquerade between the public interface ('ifpublic') and br0,&lt;br /&gt;
&amp;lt;pre&amp;gt;iptables -A FORWARD -i br0 -o ifpublic -j ACCEPT&lt;br /&gt;
iptables -A FORWARD -i ifpublic -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT&lt;br /&gt;
iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.&lt;br /&gt;
&lt;br /&gt;
== Web server ==&lt;br /&gt;
&lt;br /&gt;
Some web space must be provisioned that can serve up&lt;br /&gt;
&lt;br /&gt;
* RHCOS images&lt;br /&gt;
* Ignition files&lt;br /&gt;
&lt;br /&gt;
The system at this stage is not paranoid; In our case, I have setup a 10-openshift.conf Apache server bound to Pliny's private network interface (172.17.202.12). Stripping all commentary from the configuration file,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:80&amp;gt;&lt;br /&gt;
    ServerAdmin systems@nic.uoregon.edu&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    DocumentRoot /home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
    # if not specified, the global error log is used&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/error_log&lt;br /&gt;
    CustomLog /var/log/httpd/openshift/access_log combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        options +indexes&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:443&amp;gt;&lt;br /&gt;
    #  General setup for the virtual host&lt;br /&gt;
    DocumentRoot &amp;quot;/home/web_openshift/&amp;quot;&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/ssl_error_log&lt;br /&gt;
    TransferLog /var/log/httpd/openshift/ssl_access_log&lt;br /&gt;
&lt;br /&gt;
    SSLEngine on&lt;br /&gt;
&lt;br /&gt;
    #   SSL Protocol Support:&lt;br /&gt;
    SSLProtocol All -SSLv2 -SSLv3&lt;br /&gt;
    SSLCipherSuite    ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA&lt;br /&gt;
    SSLHonorCipherOrder     on&lt;br /&gt;
&lt;br /&gt;
    SSLCertificateFile **********&lt;br /&gt;
    SSLCertificateKeyFile **********&lt;br /&gt;
    SSLCertificateChainFile **********&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Files ~ &amp;quot;\.(cgi|shtml|phtml|php3?)$&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Files&amp;gt;&lt;br /&gt;
    &amp;lt;Directory &amp;quot;/var/www/www/cgi-bin&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Directory&amp;gt;&lt;br /&gt;
&lt;br /&gt;
    SetEnvIf User-Agent &amp;quot;.*MSIE.*&amp;quot; \&lt;br /&gt;
         nokeepalive ssl-unclean-shutdown \&lt;br /&gt;
         downgrade-1.0 force-response-1.0&lt;br /&gt;
&lt;br /&gt;
    CustomLog /var/log/httpd/ssl_request_log   ssl_combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html index.htm index.php&lt;br /&gt;
    Options +indexes&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I have decided to at least organize by processor architecture (a cluster must be 100% x86_64 or ppc64le, no mixing). The basic data that must be present in the web directory is as follows,&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
[root@pliny web_openshift]# ls /home/web_openshift/&lt;br /&gt;
openshift_ppc64le&lt;br /&gt;
[root@pliny web_openshift]# ls -la /home/web_openshift/openshift_ppc64le/&lt;br /&gt;
total 1787720&lt;br /&gt;
drwxr-xr-x. 3 root root      4096 Jul 27 19:48 .&lt;br /&gt;
drwxr-xr-x. 3 root root        31 Jul 27 19:07 ..&lt;br /&gt;
-rw-r--r--. 1 root root    288355 Jul 27 19:06 bootstrap.ign&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 master.ign&lt;br /&gt;
drwxr-xr-x. 2 root root        63 Jul 27 19:06 old&lt;br /&gt;
-rw-r--r--. 1 root root  80882648 Dec 15  2020 rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root  26903229 Dec 15  2020 rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
-rw-r--r--. 1 root root 918583296 Dec 15  2020 rhcos-4.6.8-ppc64le-live.ppc64le.iso&lt;br /&gt;
-rw-r--r--. 1 root root 803940864 Dec 15  2020 rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root       354 Jul 27 19:08 .treeinfo&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 worker.ign&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Note that the .ign files generated by ''openshift_install'' are, by default, chmod 600. If these are copied to the web server without resetting to 644 (go+r), it will fail to serve them up and the installer will explode on the launchpad without generating useful error output.''' My prepare_install.sh script performs this change automatically if used.&lt;br /&gt;
&lt;br /&gt;
Certain information must be present in the .treeinfo file:&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@pliny openshift_ppc64le]# cat .treeinfo &lt;br /&gt;
[general]&lt;br /&gt;
name = CentOS-7&lt;br /&gt;
family = CentOS&lt;br /&gt;
timestamp = 1587405659.3&lt;br /&gt;
variant =&lt;br /&gt;
version = 7&lt;br /&gt;
packagedir =&lt;br /&gt;
arch = ppc64le&lt;br /&gt;
&lt;br /&gt;
[stage2]&lt;br /&gt;
mainimage = rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
&lt;br /&gt;
[images-ppc64le]&lt;br /&gt;
kernel = rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
initrd = rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
boot.iso = rhcos-4.6.8-ppc64le-live.ppc64le.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Of course, the version given (4.6.8 here) needs to match the files actually present.&lt;br /&gt;
&lt;br /&gt;
== DNS setup ==&lt;br /&gt;
&lt;br /&gt;
Our DNS server identifies several private TLDs, including one (.stor) for the private ethernet interfaces of our nodes, for OACISS-local IP addresses. As our openshift setup will not be publicly reachable, we make it live entirely on this private TLD.&lt;br /&gt;
&lt;br /&gt;
First, we create for it the ''openshift.stor'' domain within the DNS private view,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;zone &amp;quot;openshift.stor&amp;quot; in {&lt;br /&gt;
        type master;&lt;br /&gt;
        masterfile-format text;&lt;br /&gt;
        file &amp;quot;openshift_forward.zone&amp;quot;;&lt;br /&gt;
        allow-update { none; };&lt;br /&gt;
        allow-transfer { private_servers; };&lt;br /&gt;
        allow-query { trusted_nets; };&lt;br /&gt;
};&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As we can only define one reverse lookup table for the 172.17.0.0/16 network, the reverse records must go under the storage.reverse file.&lt;br /&gt;
&lt;br /&gt;
The openshift_forward.zone file:&lt;br /&gt;
&amp;lt;pre&amp;gt;$TTL 2h&lt;br /&gt;
@                       IN      SOA     ns.nic.local. systems.nic.uoregon.edu. (&lt;br /&gt;
                                        2021072706      ; Serial number&lt;br /&gt;
                                        21600           ; Refresh(6hrs)&lt;br /&gt;
                                        1800            ; Retry(30min)&lt;br /&gt;
                                        1209600         ; Expire(2wks)&lt;br /&gt;
                                        432000 )        ; Minimum(5dys)&lt;br /&gt;
; vim: ts=4:&lt;br /&gt;
; Name servers.&lt;br /&gt;
&lt;br /&gt;
                        IN      NS      fripp.nic.local.&lt;br /&gt;
&lt;br /&gt;
; Openshift virtual machines&lt;br /&gt;
client                  IN      A       172.17.100.100&lt;br /&gt;
api                     IN      A       172.17.100.100&lt;br /&gt;
api-int                 IN      A       172.17.100.100&lt;br /&gt;
*.apps                  IN      A       172.17.100.100&lt;br /&gt;
&lt;br /&gt;
bootstrap               IN      A       172.17.100.110&lt;br /&gt;
&lt;br /&gt;
master1                 IN      A       172.17.100.101&lt;br /&gt;
master2                 IN      A       172.17.100.102&lt;br /&gt;
master3                 IN      A       172.17.100.103&lt;br /&gt;
worker1                 IN      A       172.17.100.104&lt;br /&gt;
worker2                 IN      A       172.17.100.105&lt;br /&gt;
worker3                 IN      A       172.17.100.106&lt;br /&gt;
worker4                 IN      A       172.17.100.107&lt;br /&gt;
worker5                 IN      A       172.17.100.108&lt;br /&gt;
worker6                 IN      A       172.17.100.109&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
And the relevant entries in storage_reverse.zone:&lt;br /&gt;
&amp;lt;pre&amp;gt;; OpenSHIFT virtual machine reverse lookups&lt;br /&gt;
101.100         IN      PTR     master1.openshift.stor.&lt;br /&gt;
102.100         IN      PTR     master2.openshift.stor.&lt;br /&gt;
103.100         IN      PTR     master3.openshift.stor.&lt;br /&gt;
&lt;br /&gt;
104.100         IN      PTR     worker1.openshift.stor.&lt;br /&gt;
105.100         IN      PTR     worker2.openshift.stor.&lt;br /&gt;
106.100         IN      PTR     worker3.openshift.stor.&lt;br /&gt;
107.100         IN      PTR     worker4.openshift.stor.&lt;br /&gt;
108.100         IN      PTR     worker5.openshift.stor.&lt;br /&gt;
109.100         IN      PTR     worker6.openshift.stor.&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, as is easily forgotten, that the reverse lookups must terminate with .stor. and not just .stor or reverse resolution does not work the way you expect it to :)&lt;br /&gt;
&lt;br /&gt;
The above establishes hostnames for three masters in a quorum and provisions hostnames for up to six workers.&lt;br /&gt;
&lt;br /&gt;
Do not forget to increment the dns serial numbers when this is edited!&lt;br /&gt;
&lt;br /&gt;
== Client VM ==&lt;br /&gt;
&lt;br /&gt;
As the initial entry in the OpenShift saga, we create a virtual machine named (cunningly) ''client'' from a pulled down Centos 8.3 live dvd image,&lt;br /&gt;
&amp;lt;pre&amp;gt;virt-install --virt-type=kvm --name client --memory 2048 --vcpus=2 --os-variant=rhel8.3 --cdrom=/var/lib/libvirt/boot/CentOS-8.3.2011-ppc64le-dvd1.iso --network=network=ocp,model=virtio --console=pty,target_type=virtio --disk path=/var/lib/libvirt/images/centos8.qcow2,size=20,bus=virtio,format=qcow2 --serial pty --graphics none&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will be the access point for the cluster as well as (in our case) running the load balancer.&lt;br /&gt;
&lt;br /&gt;
Before going any further, best to setup the ssh key as this will be needed shortly:&lt;br /&gt;
&amp;lt;pre&amp;gt;ssh-keygen  -t ed25519&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Load balancer ===&lt;br /&gt;
&lt;br /&gt;
OpenShift requires some kind of load balancer to provide a central access point that mediates API access to the worker nodes. We install the HAProxy load balancer onto the client VM.&lt;br /&gt;
&lt;br /&gt;
In higher performance situations, this balancer would be its own (more powerful) VM, its own machine, or in a large-scale datacenter a piece of expensive hardware.&lt;br /&gt;
&lt;br /&gt;
Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The /etc/haproxy/haproxy.cfg file we use:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#---------------------------------------------------------------------&lt;br /&gt;
# Example configuration for a possible web application.  See the&lt;br /&gt;
# full configuration options online.&lt;br /&gt;
#&lt;br /&gt;
#   https://www.haproxy.org/download/1.8/doc/configuration.txt&lt;br /&gt;
#&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# Global settings&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
global&lt;br /&gt;
    # to have these messages end up in /var/log/haproxy.log you will&lt;br /&gt;
    # need to:&lt;br /&gt;
    #&lt;br /&gt;
    # 1) configure syslog to accept network log events.  This is done&lt;br /&gt;
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in&lt;br /&gt;
    #    /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    # 2) configure local2 events to go to the /var/log/haproxy.log&lt;br /&gt;
    #   file. A line like the following can be added to&lt;br /&gt;
    #   /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    #    local2.*                       /var/log/haproxy.log&lt;br /&gt;
    #&lt;br /&gt;
    log         127.0.0.1 local2&lt;br /&gt;
&lt;br /&gt;
    chroot      /var/lib/haproxy&lt;br /&gt;
    pidfile     /var/run/haproxy.pid&lt;br /&gt;
    maxconn     4000&lt;br /&gt;
    user        haproxy&lt;br /&gt;
    group       haproxy&lt;br /&gt;
    daemon&lt;br /&gt;
&lt;br /&gt;
    # turn on stats unix socket&lt;br /&gt;
    stats socket /var/lib/haproxy/stats&lt;br /&gt;
&lt;br /&gt;
    # utilize system-wide crypto-policies&lt;br /&gt;
    ssl-default-bind-ciphers PROFILE=SYSTEM&lt;br /&gt;
    ssl-default-server-ciphers PROFILE=SYSTEM&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# common defaults that all the 'listen' and 'backend' sections will&lt;br /&gt;
# use if not designated in their block&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
defaults&lt;br /&gt;
    log                     global&lt;br /&gt;
    option                  httplog&lt;br /&gt;
    option                  dontlognull&lt;br /&gt;
    option http-server-close&lt;br /&gt;
    option forwardfor       except 127.0.0.0/8&lt;br /&gt;
    option                  redispatch&lt;br /&gt;
    retries                 3&lt;br /&gt;
    timeout http-request    10s&lt;br /&gt;
    timeout queue           1m&lt;br /&gt;
    timeout connect         10s&lt;br /&gt;
    timeout client          30s&lt;br /&gt;
    timeout server          30s&lt;br /&gt;
    timeout http-keep-alive 10s&lt;br /&gt;
    timeout check           10s&lt;br /&gt;
    maxconn                 3000&lt;br /&gt;
&lt;br /&gt;
frontend kubernetes_api&lt;br /&gt;
    bind 172.17.100.100:6443&lt;br /&gt;
    default_backend kubernetes_api&lt;br /&gt;
&lt;br /&gt;
backend kubernetes_api&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:6443 check&lt;br /&gt;
    server master1 master1.openshift.stor:6443 check&lt;br /&gt;
    server master2 master2.openshift.stor:6443 check&lt;br /&gt;
    server master3 master3.openshift.stor:6443 check&lt;br /&gt;
&lt;br /&gt;
frontend machine_config&lt;br /&gt;
    bind 172.17.100.100:22623&lt;br /&gt;
    default_backend machine_config&lt;br /&gt;
&lt;br /&gt;
backend machine_config&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:22623 check&lt;br /&gt;
    server master1 master1.openshift.stor:22623 check&lt;br /&gt;
    server master2 master2.openshift.stor:22623 check&lt;br /&gt;
    server master3 master3.openshift.stor:22623 check&lt;br /&gt;
&lt;br /&gt;
frontend router_https&lt;br /&gt;
    bind 172.17.100.100:443&lt;br /&gt;
    default_backend router_https&lt;br /&gt;
&lt;br /&gt;
backend router_https&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server worker1 worker1.openshift.stor:443 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:443 check&lt;br /&gt;
&lt;br /&gt;
frontend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    option httplog&lt;br /&gt;
    bind 172.17.100.100:80&lt;br /&gt;
    default_backend router_http&lt;br /&gt;
&lt;br /&gt;
backend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    server worker1 worker1.openshift.stor:80 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:80 check&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SElinux and firewall compatibility instructions are mandatory at this juncture:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;semanage  port -a -t http_port_t -p tcp 22623&lt;br /&gt;
semanage  port -a -t http_port_t -p tcp 6443&lt;br /&gt;
firewall-cmd --add-port=6443/tcp&lt;br /&gt;
firewall-cmd --add-port=22623/tcp&lt;br /&gt;
firewall-cmd --runtime-to-permanent&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, this assumes (correctly in the OACISS case) that the client.openshift.stor VM has only a single interface which it default places in the 'public' zone.&lt;br /&gt;
&lt;br /&gt;
= Setup process =&lt;br /&gt;
&lt;br /&gt;
== Ignition files ==&lt;br /&gt;
&lt;br /&gt;
After downloading &amp;lt;pre&amp;gt;openshift-client-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and &amp;lt;pre&amp;gt;openshift-install-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and unpacking them in /root/OCP/ on the client, it is time to generate the Ignition files that will automagically configure the virtual machines.&lt;br /&gt;
&lt;br /&gt;
These are created by openshift-install after reading a .yaml configuration file.&lt;br /&gt;
&lt;br /&gt;
This is the yaml given on the openshift install site as an example of a minimal configuration. Because we are installing the workers manually we must state 0 replicas for workers. Note that openshift-install helpfully deletes the input configuration yaml file, so this backup must be copied each time it is run...&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat install-config.yaml.bak &lt;br /&gt;
apiVersion: v1&lt;br /&gt;
baseDomain: stor&lt;br /&gt;
compute:&lt;br /&gt;
- hyperthreading: Enabled&lt;br /&gt;
  name: worker&lt;br /&gt;
  replicas: 0&lt;br /&gt;
controlPlane:&lt;br /&gt;
  hyperthreading: Enabled&lt;br /&gt;
  name: master&lt;br /&gt;
  replicas: 3&lt;br /&gt;
metadata:&lt;br /&gt;
  name: openshift&lt;br /&gt;
networking:&lt;br /&gt;
  clusterNetwork:&lt;br /&gt;
  - cidr: 10.128.0.0/14&lt;br /&gt;
    hostPrefix: 23&lt;br /&gt;
  networkType: OpenShiftSDN&lt;br /&gt;
  serviceNetwork:&lt;br /&gt;
  - 172.30.0.0/16&lt;br /&gt;
platform:&lt;br /&gt;
  none: {}&lt;br /&gt;
fips: false&lt;br /&gt;
pullSecret: '{}'&lt;br /&gt;
sshKey: 'ssh-ed25519 AAAA************ root@client.openshift.stor'&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Naturally, of course, the actual pullSecret '''and client SSH root trust key should be present.'''&lt;br /&gt;
&lt;br /&gt;
I have packaged the sequence of steps required next into a helpful prepare_install.sh script,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat prepare_install.sh p&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Deleting existing install logs and ign files&amp;quot;&lt;br /&gt;
rm -rf .openshift_install.log .openshift_install_state.json metadata.json bootstrap.ign worker.ign master.ign auth&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;cp install-config.yaml.bak install-config.yaml&amp;quot;&lt;br /&gt;
cp install-config.yaml.bak install-config.yaml&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating manifests&amp;quot;&lt;br /&gt;
./openshift-install create manifests --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating Ignition config files&amp;quot;&lt;br /&gt;
./openshift-install create ignition-configs --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Copying to web server on Pliny&amp;quot;&lt;br /&gt;
chmod go+r *ign&lt;br /&gt;
scp *ign root@pliny:/home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
cp -f /root/OCP/auth/kubeconfig /root/.kube/config&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Install bootstrap node ==&lt;br /&gt;
&lt;br /&gt;
Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;vina# virt-install \&lt;br /&gt;
--name bootstrap \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/bootstrap.qcow2,size=20,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12:80/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It will take something like 4 minutes for this to run and crank the bootstrap machine.&lt;br /&gt;
&lt;br /&gt;
Once this is done, ssh from client to core@bootstrap.openshift.stor and run 'journalctl | grep -i expired', and hopefully no output appears.&lt;br /&gt;
&lt;br /&gt;
If this succeeds, proceed.&lt;br /&gt;
&lt;br /&gt;
== Install master nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the bootstrap node is online, we can initiate installation of the master nodes,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
&lt;br /&gt;
virt-install \&lt;br /&gt;
--name master$NODE \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/master$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The $NODE substitution increments the virtual disk name, IP and hostname appropriately. We, here, have just the three master nodes as 100.101, 2 and 3.&lt;br /&gt;
&lt;br /&gt;
These should take 5-ish minutes to install, boot, self-update and reboot before they try and contact the hive mind.&lt;br /&gt;
&lt;br /&gt;
Several problems can manifest at this point, all caused by misconfiguration on the load balancer,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   88.232297] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&lt;br /&gt;
[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The problem lies in the haproxy configuration file. Under 'global' do not have 'mode http'. This is given correctly in the haproxy config above.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (31min 30s / no limit)[ 1893.903489] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #381&lt;br /&gt;
[ 1893.921039] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: x509: certificate signed by unknown authority&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This occurred when, in initially configuring haproxy, I accidentally told the forwarder for port 22623 to talk to master*:6443.&lt;br /&gt;
&lt;br /&gt;
Another possible problem is&lt;br /&gt;
&amp;lt;pre&amp;gt;[***   ] A start job is running for Ignition (fetch) (1min 45s / no limit)[  108.236817] ignition[690]: GET https://api-int.openshift.stor:22623/config/master: attempt #25&lt;br /&gt;
[  108.249306] ignition[690]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: dial tcp 172.17.100.100:22623: connect: no route to host&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This error is provoked for several possible reasons relating to the load balancer setup (See load balancer section). Most likely either&lt;br /&gt;
* Balancer not running [TCP/SYN rejected]&lt;br /&gt;
* Firewall misconfigured [TCP/SYN being dropped]&lt;br /&gt;
&lt;br /&gt;
== Install completion and bootstrap shutdown ==&lt;br /&gt;
&lt;br /&gt;
From the OCP directory on the client, once the master nodes launch into their self-setup process, run&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;./openshift-install --dir=./ wait-for bootstrap-complete --log-level=info&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This command will block your terminal window until the master nodes finish setting themselves up (a 10+ minute endeavour in my case), thereupon informing you it is safe to shut the bootstrap machine down; &amp;lt;pre&amp;gt;virsh shutdown bootstrap&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, running &amp;lt;pre&amp;gt;[root@client OCP]# ./oc get co&amp;lt;/pre&amp;gt; should vomit out 30 or so lines of the form &amp;quot;NAME [same version] True ...&amp;quot;.&lt;br /&gt;
&lt;br /&gt;
If we get nodes, we now (technically) have a working cluster,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Worker install ==&lt;br /&gt;
&lt;br /&gt;
Now it is time to install the worker nodes. This may be done on the same machine, or across whatever real machines are going to run the cluster,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
virt-install \&lt;br /&gt;
--name worker$NODE \&lt;br /&gt;
--vcpus 32 \&lt;br /&gt;
--ram 65536 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/worker$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Like the install commands for master nodes, the commands for the workers are clones, save for incrementing the IP addresses, workerN hostnames and the virtual disk names.&lt;br /&gt;
&lt;br /&gt;
== Worker install on IBM s924 ==&lt;br /&gt;
&lt;br /&gt;
First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).&lt;br /&gt;
&lt;br /&gt;
Then we must ssh to the VIOS and&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup_env&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
scp erik-k@172.17.202.18:~/downloads/rhcos-4.6.8-ppc64le-live.ppc64le.iso ./&amp;lt;/pre&amp;gt;&lt;br /&gt;
to copy the rhcos installer image to the vios, then use the system -&amp;gt; virtual storage -&amp;gt; vios -&amp;gt; manage -&amp;gt; virtual optical media to add it and assign to the rhcos LPAR.&lt;br /&gt;
&lt;br /&gt;
Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS installer screen.&lt;br /&gt;
&lt;br /&gt;
Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to terminal to substitute:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.&lt;br /&gt;
&lt;br /&gt;
== Approve new nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the consoles for the worker nodes are sitting at the login prompt, we can add them to the cluster.&lt;br /&gt;
&lt;br /&gt;
Running &amp;lt;pre&amp;gt;./oc get csr&amp;lt;/pre&amp;gt; will show that we have two key requests waiting from the workers.&lt;br /&gt;
&lt;br /&gt;
Do &amp;lt;pre&amp;gt;./oc adm certificate approve $NAME&amp;lt;/pre&amp;gt; for each of the two NAMEd requests to inject the nanoprobes and make them part of the collective.&lt;br /&gt;
&lt;br /&gt;
After about 10-15 seconds we can get nodes again and see they have appeared,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker          109s   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker          102s   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
We note that the master nodes are schedulable because we originally set ourselves up with no worker nodes. Running&lt;br /&gt;
&amp;lt;pre&amp;gt;./oc edit schedulers.config.openshift.io cluster&amp;lt;/pre&amp;gt;&lt;br /&gt;
And edit the line near the bottom for master schedulable from 'true' to 'false'. Now we have what we want:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES    AGE   VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker   10m   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker   10m   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Post-install smart moves ==&lt;br /&gt;
&lt;br /&gt;
Now that it's working, another very minor not at all noteworthy thing to mention...&lt;br /&gt;
&lt;br /&gt;
If the system ever goes down for more than 24 hr, it will be impossible to restart. So, now that it's working, this would be a great time to go run 'virsh list' and then 'virsh autostart X' all the domains: client, master[1 | 2 | 3] and worker[1 | 2] to make sure they come back up even if the host restarts.&lt;br /&gt;
&lt;br /&gt;
Now sit down and pour yourself a nice scotch, you deserve it.&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3324</id>
		<title>Procedure:OpenshiftInstall</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3324"/>
		<updated>2021-12-22T08:50:56Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Worker install on IBM s924 */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page documents the long, painful and complex process of installing OpenShift in a manner that will hopefully reveal the numerous rakes in the grass and avoid a repeat of the famous Sideshow Bob scene.&lt;br /&gt;
&lt;br /&gt;
The big picture:&lt;br /&gt;
&lt;br /&gt;
asdfasdfasdf&lt;br /&gt;
&lt;br /&gt;
= Outside guidance URLs =&lt;br /&gt;
&lt;br /&gt;
* The ostensible guide: https://docs.openshift.com/container-platform/4.6/installing/installing_bare_metal/installing-bare-metal.html#installation-obtaining-installer_installing-bare-metal&lt;br /&gt;
** This reads a lot like a semi-organized stream of consciousness, simply proceeding from A to B to C with no indexing and few explanations.&lt;br /&gt;
* https://cloud.redhat.com/openshift/downloads&lt;br /&gt;
** This leads to the openshift mirror repos where the openshift_client and openshift_installer tar files are found&lt;br /&gt;
* https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos&lt;br /&gt;
** '''PLEASE NOTE''': It is absolutely ''CRITICAL'' to have the SAME versions of the client, installer and rhcos&lt;br /&gt;
** Navigate specifically to version/xx, do not just grab 'stable'.&lt;br /&gt;
* https://www.redhat.com/en/blog/installing-openshift-41-using-libvirt-and-kvm&lt;br /&gt;
** This blog page provides some useful guidance and especially a useful template for the HAProxy configuration&lt;br /&gt;
&lt;br /&gt;
I note that this document describes the installation of a specific and isolated OC cluster.&lt;br /&gt;
&lt;br /&gt;
The OpenShift client system is itself a small VM named client.openshift.stor; This client also runs the proxy/load balancer.&lt;br /&gt;
&lt;br /&gt;
= External prerequisites =&lt;br /&gt;
&lt;br /&gt;
Openshift requires several services/components external to itself in order to work. These are,&lt;br /&gt;
* Web server: for serving up open RHCOS images and Ignition files to the installer&lt;br /&gt;
* Ethernet virbr: As this instance considers the installation of the entire cluster onto VMs living on a single real host, a virtual ethernet bridge must be established to facilitate communication&lt;br /&gt;
* Ethernet NAT masquerade: The cluster lives on a private LAN, and in this case the host machine is configured to provide NAT service to the OC virtual machines&lt;br /&gt;
* DNS: Openshift requires for itself a subdomain and a certain set of forward- and reverse-defined DNS entries.&lt;br /&gt;
* Load balancer: In this case, we install HAProxy on the client system and utilize it as the balancer. Alternatively, a dedicated balancer (VM, or machine, or hardware appliance) may be used.&lt;br /&gt;
&lt;br /&gt;
== Topology Foreword ==&lt;br /&gt;
&lt;br /&gt;
For clarity: The topology of connectivity utilized is&lt;br /&gt;
&lt;br /&gt;
(insert image here)&lt;br /&gt;
&lt;br /&gt;
== Ethernet connectivity ==&lt;br /&gt;
&lt;br /&gt;
The ethernet setup required takes two steps. First, a virtual ethernet bridge (software Level 2) must be setup on each VM host's private ethernet interface. This will create a br0 interface for the virtual bridge. The hardware en___ interface will lose its IP and be slaved to the bridge, and the br0 interface will acquire the IP.&lt;br /&gt;
&lt;br /&gt;
Now, qemu-kvm virtual machines will be able to connect to the br0 bridge, which (from their perspective) is as good as being on the physical Ethernet switch.&lt;br /&gt;
&lt;br /&gt;
As the second step, an IP masquerade/NAT is established for the cluster IPs such that they are able to name the host IP as their gateway and reach the wider Internet (to download updates and containers).&lt;br /&gt;
&lt;br /&gt;
=== Bridge setup ===&lt;br /&gt;
&lt;br /&gt;
Creating the virbr is simple on an Ubuntu host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;root@vina:~# cat /etc/netplan/00-installer-config.yaml &lt;br /&gt;
# This is the network config written by 'subiquity'&lt;br /&gt;
network:&lt;br /&gt;
  ethernets:&lt;br /&gt;
    enp1s0f0:&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
    enp1s0f1:&lt;br /&gt;
#      dhcp4: true&lt;br /&gt;
      mtu: 9000&lt;br /&gt;
  version: 2&lt;br /&gt;
  bridges:&lt;br /&gt;
    br0:&lt;br /&gt;
      interfaces: [enp1s0f1]&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
      mtu: 9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that here, mtu=9000 is set because our storage/private ethernet is setup to use jumbo frames and it is quite necessary that the IP MTUs match, or magic packet loss is likely to occur; Software bridges may automatically fragment jumbo frames, but hardware will simply discard overlength packets.&lt;br /&gt;
&lt;br /&gt;
The bridging setup on RHEL can be achieved several ways. I prefer to just punch it straight into /etc/sysconfig/network-scripts/*, as in this example that sets up one of the virbrs on our infrastructure vm node,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-eno1&lt;br /&gt;
TYPE=Ethernet&lt;br /&gt;
NAME=eno1-bridge-slave&lt;br /&gt;
UUID=9e7e3a89-3358-4830-a033-0a4154c68c55&lt;br /&gt;
DEVICE=eno1&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
BRIDGE=br0&lt;br /&gt;
HWADDR=3c:ec:ef:1a:71:3e&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-br0&lt;br /&gt;
STP=no&lt;br /&gt;
BRIDGING_OPTS=priority=32768&lt;br /&gt;
TYPE=Bridge&lt;br /&gt;
PROXY_METHOD=none&lt;br /&gt;
BROWSER_ONLY=no&lt;br /&gt;
BOOTPROTO=dhcp&lt;br /&gt;
DEFROUTE=yes&lt;br /&gt;
IPV4_FAILURE_FATAL=no&lt;br /&gt;
IPV6INIT=yes&lt;br /&gt;
IPV6_AUTOCONF=yes&lt;br /&gt;
IPV6_DEFROUTE=yes&lt;br /&gt;
IPV6_FAILURE_FATAL=no&lt;br /&gt;
IPV6_ADDR_GEN_MODE=stable-privacy&lt;br /&gt;
NAME=br0&lt;br /&gt;
UUID=3549a392-12ac-4c7f-bdd5-4e86d8654ddf&lt;br /&gt;
DEVICE=br0&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
MTU=9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It may be possible to leave STP enabled, however this was found to cause problems in some settings (the interface claims to be activating forever and never enters forwarding - i.e. the host physical ethernet port did not talk STP back)&lt;br /&gt;
&lt;br /&gt;
Once the bridge has been defined, we must inform kvm about it by defining an xml, importing it and marking it to autostart in kvm.&lt;br /&gt;
&lt;br /&gt;
=== IP masquerading ===&lt;br /&gt;
&lt;br /&gt;
Assuming that routing is enabled on the host and normal routing table entries are setup, the following three lines will setup IP masquerade between the public interface ('ifpublic') and br0,&lt;br /&gt;
&amp;lt;pre&amp;gt;iptables -A FORWARD -i br0 -o ifpublic -j ACCEPT&lt;br /&gt;
iptables -A FORWARD -i ifpublic -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT&lt;br /&gt;
iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.&lt;br /&gt;
&lt;br /&gt;
== Web server ==&lt;br /&gt;
&lt;br /&gt;
Some web space must be provisioned that can serve up&lt;br /&gt;
&lt;br /&gt;
* RHCOS images&lt;br /&gt;
* Ignition files&lt;br /&gt;
&lt;br /&gt;
The system at this stage is not paranoid; In our case, I have setup a 10-openshift.conf Apache server bound to Pliny's private network interface (172.17.202.12). Stripping all commentary from the configuration file,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:80&amp;gt;&lt;br /&gt;
    ServerAdmin systems@nic.uoregon.edu&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    DocumentRoot /home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
    # if not specified, the global error log is used&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/error_log&lt;br /&gt;
    CustomLog /var/log/httpd/openshift/access_log combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        options +indexes&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:443&amp;gt;&lt;br /&gt;
    #  General setup for the virtual host&lt;br /&gt;
    DocumentRoot &amp;quot;/home/web_openshift/&amp;quot;&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/ssl_error_log&lt;br /&gt;
    TransferLog /var/log/httpd/openshift/ssl_access_log&lt;br /&gt;
&lt;br /&gt;
    SSLEngine on&lt;br /&gt;
&lt;br /&gt;
    #   SSL Protocol Support:&lt;br /&gt;
    SSLProtocol All -SSLv2 -SSLv3&lt;br /&gt;
    SSLCipherSuite    ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA&lt;br /&gt;
    SSLHonorCipherOrder     on&lt;br /&gt;
&lt;br /&gt;
    SSLCertificateFile **********&lt;br /&gt;
    SSLCertificateKeyFile **********&lt;br /&gt;
    SSLCertificateChainFile **********&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Files ~ &amp;quot;\.(cgi|shtml|phtml|php3?)$&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Files&amp;gt;&lt;br /&gt;
    &amp;lt;Directory &amp;quot;/var/www/www/cgi-bin&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Directory&amp;gt;&lt;br /&gt;
&lt;br /&gt;
    SetEnvIf User-Agent &amp;quot;.*MSIE.*&amp;quot; \&lt;br /&gt;
         nokeepalive ssl-unclean-shutdown \&lt;br /&gt;
         downgrade-1.0 force-response-1.0&lt;br /&gt;
&lt;br /&gt;
    CustomLog /var/log/httpd/ssl_request_log   ssl_combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html index.htm index.php&lt;br /&gt;
    Options +indexes&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I have decided to at least organize by processor architecture (a cluster must be 100% x86_64 or ppc64le, no mixing). The basic data that must be present in the web directory is as follows,&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
[root@pliny web_openshift]# ls /home/web_openshift/&lt;br /&gt;
openshift_ppc64le&lt;br /&gt;
[root@pliny web_openshift]# ls -la /home/web_openshift/openshift_ppc64le/&lt;br /&gt;
total 1787720&lt;br /&gt;
drwxr-xr-x. 3 root root      4096 Jul 27 19:48 .&lt;br /&gt;
drwxr-xr-x. 3 root root        31 Jul 27 19:07 ..&lt;br /&gt;
-rw-r--r--. 1 root root    288355 Jul 27 19:06 bootstrap.ign&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 master.ign&lt;br /&gt;
drwxr-xr-x. 2 root root        63 Jul 27 19:06 old&lt;br /&gt;
-rw-r--r--. 1 root root  80882648 Dec 15  2020 rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root  26903229 Dec 15  2020 rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
-rw-r--r--. 1 root root 918583296 Dec 15  2020 rhcos-4.6.8-ppc64le-live.ppc64le.iso&lt;br /&gt;
-rw-r--r--. 1 root root 803940864 Dec 15  2020 rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root       354 Jul 27 19:08 .treeinfo&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 worker.ign&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Note that the .ign files generated by ''openshift_install'' are, by default, chmod 600. If these are copied to the web server without resetting to 644 (go+r), it will fail to serve them up and the installer will explode on the launchpad without generating useful error output.''' My prepare_install.sh script performs this change automatically if used.&lt;br /&gt;
&lt;br /&gt;
Certain information must be present in the .treeinfo file:&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@pliny openshift_ppc64le]# cat .treeinfo &lt;br /&gt;
[general]&lt;br /&gt;
name = CentOS-7&lt;br /&gt;
family = CentOS&lt;br /&gt;
timestamp = 1587405659.3&lt;br /&gt;
variant =&lt;br /&gt;
version = 7&lt;br /&gt;
packagedir =&lt;br /&gt;
arch = ppc64le&lt;br /&gt;
&lt;br /&gt;
[stage2]&lt;br /&gt;
mainimage = rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
&lt;br /&gt;
[images-ppc64le]&lt;br /&gt;
kernel = rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
initrd = rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
boot.iso = rhcos-4.6.8-ppc64le-live.ppc64le.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Of course, the version given (4.6.8 here) needs to match the files actually present.&lt;br /&gt;
&lt;br /&gt;
== DNS setup ==&lt;br /&gt;
&lt;br /&gt;
Our DNS server identifies several private TLDs, including one (.stor) for the private ethernet interfaces of our nodes, for OACISS-local IP addresses. As our openshift setup will not be publicly reachable, we make it live entirely on this private TLD.&lt;br /&gt;
&lt;br /&gt;
First, we create for it the ''openshift.stor'' domain within the DNS private view,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;zone &amp;quot;openshift.stor&amp;quot; in {&lt;br /&gt;
        type master;&lt;br /&gt;
        masterfile-format text;&lt;br /&gt;
        file &amp;quot;openshift_forward.zone&amp;quot;;&lt;br /&gt;
        allow-update { none; };&lt;br /&gt;
        allow-transfer { private_servers; };&lt;br /&gt;
        allow-query { trusted_nets; };&lt;br /&gt;
};&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As we can only define one reverse lookup table for the 172.17.0.0/16 network, the reverse records must go under the storage.reverse file.&lt;br /&gt;
&lt;br /&gt;
The openshift_forward.zone file:&lt;br /&gt;
&amp;lt;pre&amp;gt;$TTL 2h&lt;br /&gt;
@                       IN      SOA     ns.nic.local. systems.nic.uoregon.edu. (&lt;br /&gt;
                                        2021072706      ; Serial number&lt;br /&gt;
                                        21600           ; Refresh(6hrs)&lt;br /&gt;
                                        1800            ; Retry(30min)&lt;br /&gt;
                                        1209600         ; Expire(2wks)&lt;br /&gt;
                                        432000 )        ; Minimum(5dys)&lt;br /&gt;
; vim: ts=4:&lt;br /&gt;
; Name servers.&lt;br /&gt;
&lt;br /&gt;
                        IN      NS      fripp.nic.local.&lt;br /&gt;
&lt;br /&gt;
; Openshift virtual machines&lt;br /&gt;
client                  IN      A       172.17.100.100&lt;br /&gt;
api                     IN      A       172.17.100.100&lt;br /&gt;
api-int                 IN      A       172.17.100.100&lt;br /&gt;
*.apps                  IN      A       172.17.100.100&lt;br /&gt;
&lt;br /&gt;
bootstrap               IN      A       172.17.100.110&lt;br /&gt;
&lt;br /&gt;
master1                 IN      A       172.17.100.101&lt;br /&gt;
master2                 IN      A       172.17.100.102&lt;br /&gt;
master3                 IN      A       172.17.100.103&lt;br /&gt;
worker1                 IN      A       172.17.100.104&lt;br /&gt;
worker2                 IN      A       172.17.100.105&lt;br /&gt;
worker3                 IN      A       172.17.100.106&lt;br /&gt;
worker4                 IN      A       172.17.100.107&lt;br /&gt;
worker5                 IN      A       172.17.100.108&lt;br /&gt;
worker6                 IN      A       172.17.100.109&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
And the relevant entries in storage_reverse.zone:&lt;br /&gt;
&amp;lt;pre&amp;gt;; OpenSHIFT virtual machine reverse lookups&lt;br /&gt;
101.100         IN      PTR     master1.openshift.stor.&lt;br /&gt;
102.100         IN      PTR     master2.openshift.stor.&lt;br /&gt;
103.100         IN      PTR     master3.openshift.stor.&lt;br /&gt;
&lt;br /&gt;
104.100         IN      PTR     worker1.openshift.stor.&lt;br /&gt;
105.100         IN      PTR     worker2.openshift.stor.&lt;br /&gt;
106.100         IN      PTR     worker3.openshift.stor.&lt;br /&gt;
107.100         IN      PTR     worker4.openshift.stor.&lt;br /&gt;
108.100         IN      PTR     worker5.openshift.stor.&lt;br /&gt;
109.100         IN      PTR     worker6.openshift.stor.&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, as is easily forgotten, that the reverse lookups must terminate with .stor. and not just .stor or reverse resolution does not work the way you expect it to :)&lt;br /&gt;
&lt;br /&gt;
The above establishes hostnames for three masters in a quorum and provisions hostnames for up to six workers.&lt;br /&gt;
&lt;br /&gt;
Do not forget to increment the dns serial numbers when this is edited!&lt;br /&gt;
&lt;br /&gt;
== Client VM ==&lt;br /&gt;
&lt;br /&gt;
As the initial entry in the OpenShift saga, we create a virtual machine named (cunningly) ''client'' from a pulled down Centos 8.3 live dvd image,&lt;br /&gt;
&amp;lt;pre&amp;gt;virt-install --virt-type=kvm --name client --memory 2048 --vcpus=2 --os-variant=rhel8.3 --cdrom=/var/lib/libvirt/boot/CentOS-8.3.2011-ppc64le-dvd1.iso --network=network=ocp,model=virtio --console=pty,target_type=virtio --disk path=/var/lib/libvirt/images/centos8.qcow2,size=20,bus=virtio,format=qcow2 --serial pty --graphics none&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will be the access point for the cluster as well as (in our case) running the load balancer.&lt;br /&gt;
&lt;br /&gt;
Before going any further, best to setup the ssh key as this will be needed shortly:&lt;br /&gt;
&amp;lt;pre&amp;gt;ssh-keygen  -t ed25519&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Load balancer ===&lt;br /&gt;
&lt;br /&gt;
We install the HAProxy load balancer onto the client VM.&lt;br /&gt;
&lt;br /&gt;
In a more performant situation, this load balancer would be loaded onto a separate (more powerful) VM, onto its own entire machine, or in true datacenter applications would be a very expensive piece of hardware.&lt;br /&gt;
&lt;br /&gt;
Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The /etc/haproxy/haproxy.cfg file:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#---------------------------------------------------------------------&lt;br /&gt;
# Example configuration for a possible web application.  See the&lt;br /&gt;
# full configuration options online.&lt;br /&gt;
#&lt;br /&gt;
#   https://www.haproxy.org/download/1.8/doc/configuration.txt&lt;br /&gt;
#&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# Global settings&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
global&lt;br /&gt;
    # to have these messages end up in /var/log/haproxy.log you will&lt;br /&gt;
    # need to:&lt;br /&gt;
    #&lt;br /&gt;
    # 1) configure syslog to accept network log events.  This is done&lt;br /&gt;
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in&lt;br /&gt;
    #    /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    # 2) configure local2 events to go to the /var/log/haproxy.log&lt;br /&gt;
    #   file. A line like the following can be added to&lt;br /&gt;
    #   /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    #    local2.*                       /var/log/haproxy.log&lt;br /&gt;
    #&lt;br /&gt;
    log         127.0.0.1 local2&lt;br /&gt;
&lt;br /&gt;
    chroot      /var/lib/haproxy&lt;br /&gt;
    pidfile     /var/run/haproxy.pid&lt;br /&gt;
    maxconn     4000&lt;br /&gt;
    user        haproxy&lt;br /&gt;
    group       haproxy&lt;br /&gt;
    daemon&lt;br /&gt;
&lt;br /&gt;
    # turn on stats unix socket&lt;br /&gt;
    stats socket /var/lib/haproxy/stats&lt;br /&gt;
&lt;br /&gt;
    # utilize system-wide crypto-policies&lt;br /&gt;
    ssl-default-bind-ciphers PROFILE=SYSTEM&lt;br /&gt;
    ssl-default-server-ciphers PROFILE=SYSTEM&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# common defaults that all the 'listen' and 'backend' sections will&lt;br /&gt;
# use if not designated in their block&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
defaults&lt;br /&gt;
    log                     global&lt;br /&gt;
    option                  httplog&lt;br /&gt;
    option                  dontlognull&lt;br /&gt;
    option http-server-close&lt;br /&gt;
    option forwardfor       except 127.0.0.0/8&lt;br /&gt;
    option                  redispatch&lt;br /&gt;
    retries                 3&lt;br /&gt;
    timeout http-request    10s&lt;br /&gt;
    timeout queue           1m&lt;br /&gt;
    timeout connect         10s&lt;br /&gt;
    timeout client          30s&lt;br /&gt;
    timeout server          30s&lt;br /&gt;
    timeout http-keep-alive 10s&lt;br /&gt;
    timeout check           10s&lt;br /&gt;
    maxconn                 3000&lt;br /&gt;
&lt;br /&gt;
frontend kubernetes_api&lt;br /&gt;
    bind 172.17.100.100:6443&lt;br /&gt;
    default_backend kubernetes_api&lt;br /&gt;
&lt;br /&gt;
backend kubernetes_api&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:6443 check&lt;br /&gt;
    server master1 master1.openshift.stor:6443 check&lt;br /&gt;
    server master2 master2.openshift.stor:6443 check&lt;br /&gt;
    server master3 master3.openshift.stor:6443 check&lt;br /&gt;
&lt;br /&gt;
frontend machine_config&lt;br /&gt;
    bind 172.17.100.100:22623&lt;br /&gt;
    default_backend machine_config&lt;br /&gt;
&lt;br /&gt;
backend machine_config&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:22623 check&lt;br /&gt;
    server master1 master1.openshift.stor:22623 check&lt;br /&gt;
    server master2 master2.openshift.stor:22623 check&lt;br /&gt;
    server master3 master3.openshift.stor:22623 check&lt;br /&gt;
&lt;br /&gt;
frontend router_https&lt;br /&gt;
    bind 172.17.100.100:443&lt;br /&gt;
    default_backend router_https&lt;br /&gt;
&lt;br /&gt;
backend router_https&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server worker1 worker1.openshift.stor:443 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:443 check&lt;br /&gt;
&lt;br /&gt;
frontend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    option httplog&lt;br /&gt;
    bind 172.17.100.100:80&lt;br /&gt;
    default_backend router_http&lt;br /&gt;
&lt;br /&gt;
backend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    server worker1 worker1.openshift.stor:80 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:80 check&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SElinux and firewall compatibility instructions are mandatory at this juncture:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;semanage  port -a -t http_port_t -p tcp 22623&lt;br /&gt;
semanage  port -a -t http_port_t -p tcp 6443&lt;br /&gt;
firewall-cmd --add-port=6443/tcp&lt;br /&gt;
firewall-cmd --add-port=22623/tcp&lt;br /&gt;
firewall-cmd --runtime-to-permanent&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, this assumes (correctly in the OACISS case) that the client.openshift.stor VM has only a single interface which it default places in the 'public' zone.&lt;br /&gt;
&lt;br /&gt;
= Setup process =&lt;br /&gt;
&lt;br /&gt;
== Ignition files ==&lt;br /&gt;
&lt;br /&gt;
After downloading &amp;lt;pre&amp;gt;openshift-client-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and &amp;lt;pre&amp;gt;openshift-install-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and unpacking them in /root/OCP/ on the client, it is time to generate the Ignition files that will automagically configure the virtual machines.&lt;br /&gt;
&lt;br /&gt;
These are created by openshift-install after reading a .yaml configuration file.&lt;br /&gt;
&lt;br /&gt;
This is the yaml given on the openshift install site as an example of a minimal configuration. Because we are installing the workers manually we must state 0 replicas for workers. Note that openshift-install helpfully deletes the input configuration yaml file, so this backup must be copied each time it is run...&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat install-config.yaml.bak &lt;br /&gt;
apiVersion: v1&lt;br /&gt;
baseDomain: stor&lt;br /&gt;
compute:&lt;br /&gt;
- hyperthreading: Enabled&lt;br /&gt;
  name: worker&lt;br /&gt;
  replicas: 0&lt;br /&gt;
controlPlane:&lt;br /&gt;
  hyperthreading: Enabled&lt;br /&gt;
  name: master&lt;br /&gt;
  replicas: 3&lt;br /&gt;
metadata:&lt;br /&gt;
  name: openshift&lt;br /&gt;
networking:&lt;br /&gt;
  clusterNetwork:&lt;br /&gt;
  - cidr: 10.128.0.0/14&lt;br /&gt;
    hostPrefix: 23&lt;br /&gt;
  networkType: OpenShiftSDN&lt;br /&gt;
  serviceNetwork:&lt;br /&gt;
  - 172.30.0.0/16&lt;br /&gt;
platform:&lt;br /&gt;
  none: {}&lt;br /&gt;
fips: false&lt;br /&gt;
pullSecret: '{}'&lt;br /&gt;
sshKey: 'ssh-ed25519 AAAA************ root@client.openshift.stor'&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Naturally, of course, the actual pullSecret '''and client SSH root trust key should be present.'''&lt;br /&gt;
&lt;br /&gt;
I have packaged the sequence of steps required next into a helpful prepare_install.sh script,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat prepare_install.sh p&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Deleting existing install logs and ign files&amp;quot;&lt;br /&gt;
rm -rf .openshift_install.log .openshift_install_state.json metadata.json bootstrap.ign worker.ign master.ign auth&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;cp install-config.yaml.bak install-config.yaml&amp;quot;&lt;br /&gt;
cp install-config.yaml.bak install-config.yaml&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating manifests&amp;quot;&lt;br /&gt;
./openshift-install create manifests --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating Ignition config files&amp;quot;&lt;br /&gt;
./openshift-install create ignition-configs --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Copying to web server on Pliny&amp;quot;&lt;br /&gt;
chmod go+r *ign&lt;br /&gt;
scp *ign root@pliny:/home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
cp -f /root/OCP/auth/kubeconfig /root/.kube/config&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Install bootstrap node ==&lt;br /&gt;
&lt;br /&gt;
Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;vina# virt-install \&lt;br /&gt;
--name bootstrap \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/bootstrap.qcow2,size=20,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12:80/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It will take something like 4 minutes for this to run and crank the bootstrap machine.&lt;br /&gt;
&lt;br /&gt;
Once this is done, ssh from client to core@bootstrap.openshift.stor and run 'journalctl | grep -i expired', and hopefully no output appears.&lt;br /&gt;
&lt;br /&gt;
If this succeeds, proceed.&lt;br /&gt;
&lt;br /&gt;
== Install master nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the bootstrap node is online, we can initiate installation of the master nodes,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
&lt;br /&gt;
virt-install \&lt;br /&gt;
--name master$NODE \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/master$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The $NODE substitution increments the virtual disk name, IP and hostname appropriately. We, here, have just the three master nodes as 100.101, 2 and 3.&lt;br /&gt;
&lt;br /&gt;
These should take 5-ish minutes to install, boot, self-update and reboot before they try and contact the hive mind.&lt;br /&gt;
&lt;br /&gt;
Several problems can manifest at this point, all caused by misconfiguration on the load balancer,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   88.232297] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&lt;br /&gt;
[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The problem lies in the haproxy configuration file. Under 'global' do not have 'mode http'. This is given correctly in the haproxy config above.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (31min 30s / no limit)[ 1893.903489] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #381&lt;br /&gt;
[ 1893.921039] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: x509: certificate signed by unknown authority&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This occurred when, in initially configuring haproxy, I accidentally told the forwarder for port 22623 to talk to master*:6443.&lt;br /&gt;
&lt;br /&gt;
Another possible problem is&lt;br /&gt;
&amp;lt;pre&amp;gt;[***   ] A start job is running for Ignition (fetch) (1min 45s / no limit)[  108.236817] ignition[690]: GET https://api-int.openshift.stor:22623/config/master: attempt #25&lt;br /&gt;
[  108.249306] ignition[690]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: dial tcp 172.17.100.100:22623: connect: no route to host&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This error is provoked for several possible reasons relating to the load balancer setup (See load balancer section). Most likely either&lt;br /&gt;
* Balancer not running [TCP/SYN rejected]&lt;br /&gt;
* Firewall misconfigured [TCP/SYN being dropped]&lt;br /&gt;
&lt;br /&gt;
== Install completion and bootstrap shutdown ==&lt;br /&gt;
&lt;br /&gt;
From the OCP directory on the client, once the master nodes launch into their self-setup process, run&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;./openshift-install --dir=./ wait-for bootstrap-complete --log-level=info&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This command will block your terminal window until the master nodes finish setting themselves up (a 10+ minute endeavour in my case), thereupon informing you it is safe to shut the bootstrap machine down; &amp;lt;pre&amp;gt;virsh shutdown bootstrap&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, running &amp;lt;pre&amp;gt;[root@client OCP]# ./oc get co&amp;lt;/pre&amp;gt; should vomit out 30 or so lines of the form &amp;quot;NAME [same version] True ...&amp;quot;.&lt;br /&gt;
&lt;br /&gt;
If we get nodes, we now (technically) have a working cluster,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Worker install ==&lt;br /&gt;
&lt;br /&gt;
Now it is time to install the worker nodes. This may be done on the same machine, or across whatever real machines are going to run the cluster,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
virt-install \&lt;br /&gt;
--name worker$NODE \&lt;br /&gt;
--vcpus 32 \&lt;br /&gt;
--ram 65536 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/worker$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Like the install commands for master nodes, the commands for the workers are clones, save for incrementing the IP addresses, workerN hostnames and the virtual disk names.&lt;br /&gt;
&lt;br /&gt;
== Worker install on IBM s924 ==&lt;br /&gt;
&lt;br /&gt;
First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).&lt;br /&gt;
&lt;br /&gt;
Then we must ssh to the VIOS and&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup_env&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
scp erik-k@172.17.202.18:~/downloads/rhcos-4.6.8-ppc64le-live.ppc64le.iso ./&amp;lt;/pre&amp;gt;&lt;br /&gt;
to copy the rhcos installer image to the vios, then use the system -&amp;gt; virtual storage -&amp;gt; vios -&amp;gt; manage -&amp;gt; virtual optical media to add it and assign to the rhcos LPAR.&lt;br /&gt;
&lt;br /&gt;
Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS installer screen.&lt;br /&gt;
&lt;br /&gt;
Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to terminal to substitute:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.&lt;br /&gt;
&lt;br /&gt;
== Approve new nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the consoles for the worker nodes are sitting at the login prompt, we can add them to the cluster.&lt;br /&gt;
&lt;br /&gt;
Running &amp;lt;pre&amp;gt;./oc get csr&amp;lt;/pre&amp;gt; will show that we have two key requests waiting from the workers.&lt;br /&gt;
&lt;br /&gt;
Do &amp;lt;pre&amp;gt;./oc adm certificate approve $NAME&amp;lt;/pre&amp;gt; for each of the two NAMEd requests to inject the nanoprobes and make them part of the collective.&lt;br /&gt;
&lt;br /&gt;
After about 10-15 seconds we can get nodes again and see they have appeared,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker          109s   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker          102s   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
We note that the master nodes are schedulable because we originally set ourselves up with no worker nodes. Running&lt;br /&gt;
&amp;lt;pre&amp;gt;./oc edit schedulers.config.openshift.io cluster&amp;lt;/pre&amp;gt;&lt;br /&gt;
And edit the line near the bottom for master schedulable from 'true' to 'false'. Now we have what we want:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES    AGE   VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker   10m   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker   10m   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Post-install smart moves ==&lt;br /&gt;
&lt;br /&gt;
Now that it's working, another very minor not at all noteworthy thing to mention...&lt;br /&gt;
&lt;br /&gt;
If the system ever goes down for more than 24 hr, it will be impossible to restart. So, now that it's working, this would be a great time to go run 'virsh list' and then 'virsh autostart X' all the domains: client, master[1 | 2 | 3] and worker[1 | 2] to make sure they come back up even if the host restarts.&lt;br /&gt;
&lt;br /&gt;
Now sit down and pour yourself a nice scotch, you deserve it.&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3323</id>
		<title>Procedure:OpenshiftInstall</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Procedure:OpenshiftInstall&amp;diff=3323"/>
		<updated>2021-12-21T22:25:17Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page documents the long, painful and complex process of installing OpenShift in a manner that will hopefully reveal the numerous rakes in the grass and avoid a repeat of the famous Sideshow Bob scene.&lt;br /&gt;
&lt;br /&gt;
The big picture:&lt;br /&gt;
&lt;br /&gt;
asdfasdfasdf&lt;br /&gt;
&lt;br /&gt;
= Outside guidance URLs =&lt;br /&gt;
&lt;br /&gt;
* The ostensible guide: https://docs.openshift.com/container-platform/4.6/installing/installing_bare_metal/installing-bare-metal.html#installation-obtaining-installer_installing-bare-metal&lt;br /&gt;
** This reads a lot like a semi-organized stream of consciousness, simply proceeding from A to B to C with no indexing and few explanations.&lt;br /&gt;
* https://cloud.redhat.com/openshift/downloads&lt;br /&gt;
** This leads to the openshift mirror repos where the openshift_client and openshift_installer tar files are found&lt;br /&gt;
* https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos&lt;br /&gt;
** '''PLEASE NOTE''': It is absolutely ''CRITICAL'' to have the SAME versions of the client, installer and rhcos&lt;br /&gt;
** Navigate specifically to version/xx, do not just grab 'stable'.&lt;br /&gt;
* https://www.redhat.com/en/blog/installing-openshift-41-using-libvirt-and-kvm&lt;br /&gt;
** This blog page provides some useful guidance and especially a useful template for the HAProxy configuration&lt;br /&gt;
&lt;br /&gt;
I note that this document describes the installation of a specific and isolated OC cluster.&lt;br /&gt;
&lt;br /&gt;
The OpenShift client system is itself a small VM named client.openshift.stor; This client also runs the proxy/load balancer.&lt;br /&gt;
&lt;br /&gt;
= External prerequisites =&lt;br /&gt;
&lt;br /&gt;
Openshift requires several services/components external to itself in order to work. These are,&lt;br /&gt;
* Web server: for serving up open RHCOS images and Ignition files to the installer&lt;br /&gt;
* Ethernet virbr: As this instance considers the installation of the entire cluster onto VMs living on a single real host, a virtual ethernet bridge must be established to facilitate communication&lt;br /&gt;
* Ethernet NAT masquerade: The cluster lives on a private LAN, and in this case the host machine is configured to provide NAT service to the OC virtual machines&lt;br /&gt;
* DNS: Openshift requires for itself a subdomain and a certain set of forward- and reverse-defined DNS entries.&lt;br /&gt;
* Load balancer: In this case, we install HAProxy on the client system and utilize it as the balancer. Alternatively, a dedicated balancer (VM, or machine, or hardware appliance) may be used.&lt;br /&gt;
&lt;br /&gt;
== Topology Foreword ==&lt;br /&gt;
&lt;br /&gt;
For clarity: The topology of connectivity utilized is&lt;br /&gt;
&lt;br /&gt;
(insert image here)&lt;br /&gt;
&lt;br /&gt;
== Ethernet connectivity ==&lt;br /&gt;
&lt;br /&gt;
The ethernet setup required takes two steps. First, a virtual ethernet bridge (software Level 2) must be setup on each VM host's private ethernet interface. This will create a br0 interface for the virtual bridge. The hardware en___ interface will lose its IP and be slaved to the bridge, and the br0 interface will acquire the IP.&lt;br /&gt;
&lt;br /&gt;
Now, qemu-kvm virtual machines will be able to connect to the br0 bridge, which (from their perspective) is as good as being on the physical Ethernet switch.&lt;br /&gt;
&lt;br /&gt;
As the second step, an IP masquerade/NAT is established for the cluster IPs such that they are able to name the host IP as their gateway and reach the wider Internet (to download updates and containers).&lt;br /&gt;
&lt;br /&gt;
=== Bridge setup ===&lt;br /&gt;
&lt;br /&gt;
Creating the virbr is simple on an Ubuntu host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;root@vina:~# cat /etc/netplan/00-installer-config.yaml &lt;br /&gt;
# This is the network config written by 'subiquity'&lt;br /&gt;
network:&lt;br /&gt;
  ethernets:&lt;br /&gt;
    enp1s0f0:&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
    enp1s0f1:&lt;br /&gt;
#      dhcp4: true&lt;br /&gt;
      mtu: 9000&lt;br /&gt;
  version: 2&lt;br /&gt;
  bridges:&lt;br /&gt;
    br0:&lt;br /&gt;
      interfaces: [enp1s0f1]&lt;br /&gt;
      dhcp4: true&lt;br /&gt;
      mtu: 9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that here, mtu=9000 is set because our storage/private ethernet is setup to use jumbo frames and it is quite necessary that the IP MTUs match, or magic packet loss is likely to occur; Software bridges may automatically fragment jumbo frames, but hardware will simply discard overlength packets.&lt;br /&gt;
&lt;br /&gt;
The bridging setup on RHEL can be achieved several ways. I prefer to just punch it straight into /etc/sysconfig/network-scripts/*, as in this example that sets up one of the virbrs on our infrastructure vm node,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-eno1&lt;br /&gt;
TYPE=Ethernet&lt;br /&gt;
NAME=eno1-bridge-slave&lt;br /&gt;
UUID=9e7e3a89-3358-4830-a033-0a4154c68c55&lt;br /&gt;
DEVICE=eno1&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
BRIDGE=br0&lt;br /&gt;
HWADDR=3c:ec:ef:1a:71:3e&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@orion network-scripts]# cat ifcfg-br0&lt;br /&gt;
STP=no&lt;br /&gt;
BRIDGING_OPTS=priority=32768&lt;br /&gt;
TYPE=Bridge&lt;br /&gt;
PROXY_METHOD=none&lt;br /&gt;
BROWSER_ONLY=no&lt;br /&gt;
BOOTPROTO=dhcp&lt;br /&gt;
DEFROUTE=yes&lt;br /&gt;
IPV4_FAILURE_FATAL=no&lt;br /&gt;
IPV6INIT=yes&lt;br /&gt;
IPV6_AUTOCONF=yes&lt;br /&gt;
IPV6_DEFROUTE=yes&lt;br /&gt;
IPV6_FAILURE_FATAL=no&lt;br /&gt;
IPV6_ADDR_GEN_MODE=stable-privacy&lt;br /&gt;
NAME=br0&lt;br /&gt;
UUID=3549a392-12ac-4c7f-bdd5-4e86d8654ddf&lt;br /&gt;
DEVICE=br0&lt;br /&gt;
ONBOOT=yes&lt;br /&gt;
MTU=9000&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It may be possible to leave STP enabled, however this was found to cause problems in some settings (the interface claims to be activating forever and never enters forwarding - i.e. the host physical ethernet port did not talk STP back)&lt;br /&gt;
&lt;br /&gt;
Once the bridge has been defined, we must inform kvm about it by defining an xml, importing it and marking it to autostart in kvm.&lt;br /&gt;
&lt;br /&gt;
=== IP masquerading ===&lt;br /&gt;
&lt;br /&gt;
Assuming that routing is enabled on the host and normal routing table entries are setup, the following three lines will setup IP masquerade between the public interface ('ifpublic') and br0,&lt;br /&gt;
&amp;lt;pre&amp;gt;iptables -A FORWARD -i br0 -o ifpublic -j ACCEPT&lt;br /&gt;
iptables -A FORWARD -i ifpublic -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT&lt;br /&gt;
iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.&lt;br /&gt;
&lt;br /&gt;
== Web server ==&lt;br /&gt;
&lt;br /&gt;
Some web space must be provisioned that can serve up&lt;br /&gt;
&lt;br /&gt;
* RHCOS images&lt;br /&gt;
* Ignition files&lt;br /&gt;
&lt;br /&gt;
The system at this stage is not paranoid; In our case, I have setup a 10-openshift.conf Apache server bound to Pliny's private network interface (172.17.202.12). Stripping all commentary from the configuration file,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:80&amp;gt;&lt;br /&gt;
    ServerAdmin systems@nic.uoregon.edu&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    DocumentRoot /home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
    # if not specified, the global error log is used&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/error_log&lt;br /&gt;
    CustomLog /var/log/httpd/openshift/access_log combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        options +indexes&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;VirtualHost 172.17.202.12:443&amp;gt;&lt;br /&gt;
    #  General setup for the virtual host&lt;br /&gt;
    DocumentRoot &amp;quot;/home/web_openshift/&amp;quot;&lt;br /&gt;
    ServerName pliny.nic.uoregon.edu&lt;br /&gt;
&lt;br /&gt;
    ErrorLog /var/log/httpd/openshift/ssl_error_log&lt;br /&gt;
    TransferLog /var/log/httpd/openshift/ssl_access_log&lt;br /&gt;
&lt;br /&gt;
    SSLEngine on&lt;br /&gt;
&lt;br /&gt;
    #   SSL Protocol Support:&lt;br /&gt;
    SSLProtocol All -SSLv2 -SSLv3&lt;br /&gt;
    SSLCipherSuite    ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA&lt;br /&gt;
    SSLHonorCipherOrder     on&lt;br /&gt;
&lt;br /&gt;
    SSLCertificateFile **********&lt;br /&gt;
    SSLCertificateKeyFile **********&lt;br /&gt;
    SSLCertificateChainFile **********&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Files ~ &amp;quot;\.(cgi|shtml|phtml|php3?)$&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Files&amp;gt;&lt;br /&gt;
    &amp;lt;Directory &amp;quot;/var/www/www/cgi-bin&amp;quot;&amp;gt;&lt;br /&gt;
        SSLOptions +StdEnvVars&lt;br /&gt;
    &amp;lt;/Directory&amp;gt;&lt;br /&gt;
&lt;br /&gt;
    SetEnvIf User-Agent &amp;quot;.*MSIE.*&amp;quot; \&lt;br /&gt;
         nokeepalive ssl-unclean-shutdown \&lt;br /&gt;
         downgrade-1.0 force-response-1.0&lt;br /&gt;
&lt;br /&gt;
    CustomLog /var/log/httpd/ssl_request_log   ssl_combined&lt;br /&gt;
&lt;br /&gt;
    HostnameLookups Off&lt;br /&gt;
    UseCanonicalName Off&lt;br /&gt;
    ServerSignature On&lt;br /&gt;
    DirectoryIndex index.html index.htm index.php&lt;br /&gt;
    Options +indexes&lt;br /&gt;
&lt;br /&gt;
    &amp;lt;Location /&amp;gt;&lt;br /&gt;
        &amp;lt;RequireAny&amp;gt;&lt;br /&gt;
            Require ip 172.17.0.0/16&lt;br /&gt;
        &amp;lt;/RequireAny&amp;gt;&lt;br /&gt;
    &amp;lt;/Location&amp;gt;&lt;br /&gt;
&amp;lt;/VirtualHost&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I have decided to at least organize by processor architecture (a cluster must be 100% x86_64 or ppc64le, no mixing). The basic data that must be present in the web directory is as follows,&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
[root@pliny web_openshift]# ls /home/web_openshift/&lt;br /&gt;
openshift_ppc64le&lt;br /&gt;
[root@pliny web_openshift]# ls -la /home/web_openshift/openshift_ppc64le/&lt;br /&gt;
total 1787720&lt;br /&gt;
drwxr-xr-x. 3 root root      4096 Jul 27 19:48 .&lt;br /&gt;
drwxr-xr-x. 3 root root        31 Jul 27 19:07 ..&lt;br /&gt;
-rw-r--r--. 1 root root    288355 Jul 27 19:06 bootstrap.ign&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 master.ign&lt;br /&gt;
drwxr-xr-x. 2 root root        63 Jul 27 19:06 old&lt;br /&gt;
-rw-r--r--. 1 root root  80882648 Dec 15  2020 rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root  26903229 Dec 15  2020 rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
-rw-r--r--. 1 root root 918583296 Dec 15  2020 rhcos-4.6.8-ppc64le-live.ppc64le.iso&lt;br /&gt;
-rw-r--r--. 1 root root 803940864 Dec 15  2020 rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
-rw-r--r--. 1 root root       354 Jul 27 19:08 .treeinfo&lt;br /&gt;
-rw-r--r--. 1 root root      1716 Jul 27 19:06 worker.ign&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Note that the .ign files generated by ''openshift_install'' are, by default, chmod 600. If these are copied to the web server without resetting to 644 (go+r), it will fail to serve them up and the installer will explode on the launchpad without generating useful error output.''' My prepare_install.sh script performs this change automatically if used.&lt;br /&gt;
&lt;br /&gt;
Certain information must be present in the .treeinfo file:&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@pliny openshift_ppc64le]# cat .treeinfo &lt;br /&gt;
[general]&lt;br /&gt;
name = CentOS-7&lt;br /&gt;
family = CentOS&lt;br /&gt;
timestamp = 1587405659.3&lt;br /&gt;
variant =&lt;br /&gt;
version = 7&lt;br /&gt;
packagedir =&lt;br /&gt;
arch = ppc64le&lt;br /&gt;
&lt;br /&gt;
[stage2]&lt;br /&gt;
mainimage = rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img&lt;br /&gt;
&lt;br /&gt;
[images-ppc64le]&lt;br /&gt;
kernel = rhcos-4.6.8-ppc64le-live-kernel-ppc64le&lt;br /&gt;
initrd = rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img&lt;br /&gt;
boot.iso = rhcos-4.6.8-ppc64le-live.ppc64le.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Of course, the version given (4.6.8 here) needs to match the files actually present.&lt;br /&gt;
&lt;br /&gt;
== DNS setup ==&lt;br /&gt;
&lt;br /&gt;
Our DNS server identifies several private TLDs, including one (.stor) for the private ethernet interfaces of our nodes, for OACISS-local IP addresses. As our openshift setup will not be publicly reachable, we make it live entirely on this private TLD.&lt;br /&gt;
&lt;br /&gt;
First, we create for it the ''openshift.stor'' domain within the DNS private view,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;zone &amp;quot;openshift.stor&amp;quot; in {&lt;br /&gt;
        type master;&lt;br /&gt;
        masterfile-format text;&lt;br /&gt;
        file &amp;quot;openshift_forward.zone&amp;quot;;&lt;br /&gt;
        allow-update { none; };&lt;br /&gt;
        allow-transfer { private_servers; };&lt;br /&gt;
        allow-query { trusted_nets; };&lt;br /&gt;
};&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
As we can only define one reverse lookup table for the 172.17.0.0/16 network, the reverse records must go under the storage.reverse file.&lt;br /&gt;
&lt;br /&gt;
The openshift_forward.zone file:&lt;br /&gt;
&amp;lt;pre&amp;gt;$TTL 2h&lt;br /&gt;
@                       IN      SOA     ns.nic.local. systems.nic.uoregon.edu. (&lt;br /&gt;
                                        2021072706      ; Serial number&lt;br /&gt;
                                        21600           ; Refresh(6hrs)&lt;br /&gt;
                                        1800            ; Retry(30min)&lt;br /&gt;
                                        1209600         ; Expire(2wks)&lt;br /&gt;
                                        432000 )        ; Minimum(5dys)&lt;br /&gt;
; vim: ts=4:&lt;br /&gt;
; Name servers.&lt;br /&gt;
&lt;br /&gt;
                        IN      NS      fripp.nic.local.&lt;br /&gt;
&lt;br /&gt;
; Openshift virtual machines&lt;br /&gt;
client                  IN      A       172.17.100.100&lt;br /&gt;
api                     IN      A       172.17.100.100&lt;br /&gt;
api-int                 IN      A       172.17.100.100&lt;br /&gt;
*.apps                  IN      A       172.17.100.100&lt;br /&gt;
&lt;br /&gt;
bootstrap               IN      A       172.17.100.110&lt;br /&gt;
&lt;br /&gt;
master1                 IN      A       172.17.100.101&lt;br /&gt;
master2                 IN      A       172.17.100.102&lt;br /&gt;
master3                 IN      A       172.17.100.103&lt;br /&gt;
worker1                 IN      A       172.17.100.104&lt;br /&gt;
worker2                 IN      A       172.17.100.105&lt;br /&gt;
worker3                 IN      A       172.17.100.106&lt;br /&gt;
worker4                 IN      A       172.17.100.107&lt;br /&gt;
worker5                 IN      A       172.17.100.108&lt;br /&gt;
worker6                 IN      A       172.17.100.109&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
And the relevant entries in storage_reverse.zone:&lt;br /&gt;
&amp;lt;pre&amp;gt;; OpenSHIFT virtual machine reverse lookups&lt;br /&gt;
101.100         IN      PTR     master1.openshift.stor.&lt;br /&gt;
102.100         IN      PTR     master2.openshift.stor.&lt;br /&gt;
103.100         IN      PTR     master3.openshift.stor.&lt;br /&gt;
&lt;br /&gt;
104.100         IN      PTR     worker1.openshift.stor.&lt;br /&gt;
105.100         IN      PTR     worker2.openshift.stor.&lt;br /&gt;
106.100         IN      PTR     worker3.openshift.stor.&lt;br /&gt;
107.100         IN      PTR     worker4.openshift.stor.&lt;br /&gt;
108.100         IN      PTR     worker5.openshift.stor.&lt;br /&gt;
109.100         IN      PTR     worker6.openshift.stor.&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, as is easily forgotten, that the reverse lookups must terminate with .stor. and not just .stor or reverse resolution does not work the way you expect it to :)&lt;br /&gt;
&lt;br /&gt;
The above establishes hostnames for three masters in a quorum and provisions hostnames for up to six workers.&lt;br /&gt;
&lt;br /&gt;
Do not forget to increment the dns serial numbers when this is edited!&lt;br /&gt;
&lt;br /&gt;
== Client VM ==&lt;br /&gt;
&lt;br /&gt;
As the initial entry in the OpenShift saga, we create a virtual machine named (cunningly) ''client'' from a pulled down Centos 8.3 live dvd image,&lt;br /&gt;
&amp;lt;pre&amp;gt;virt-install --virt-type=kvm --name client --memory 2048 --vcpus=2 --os-variant=rhel8.3 --cdrom=/var/lib/libvirt/boot/CentOS-8.3.2011-ppc64le-dvd1.iso --network=network=ocp,model=virtio --console=pty,target_type=virtio --disk path=/var/lib/libvirt/images/centos8.qcow2,size=20,bus=virtio,format=qcow2 --serial pty --graphics none&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will be the access point for the cluster as well as (in our case) running the load balancer.&lt;br /&gt;
&lt;br /&gt;
Before going any further, best to setup the ssh key as this will be needed shortly:&lt;br /&gt;
&amp;lt;pre&amp;gt;ssh-keygen  -t ed25519&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Load balancer ===&lt;br /&gt;
&lt;br /&gt;
We install the HAProxy load balancer onto the client VM.&lt;br /&gt;
&lt;br /&gt;
In a more performant situation, this load balancer would be loaded onto a separate (more powerful) VM, onto its own entire machine, or in true datacenter applications would be a very expensive piece of hardware.&lt;br /&gt;
&lt;br /&gt;
Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The /etc/haproxy/haproxy.cfg file:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;#---------------------------------------------------------------------&lt;br /&gt;
# Example configuration for a possible web application.  See the&lt;br /&gt;
# full configuration options online.&lt;br /&gt;
#&lt;br /&gt;
#   https://www.haproxy.org/download/1.8/doc/configuration.txt&lt;br /&gt;
#&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# Global settings&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
global&lt;br /&gt;
    # to have these messages end up in /var/log/haproxy.log you will&lt;br /&gt;
    # need to:&lt;br /&gt;
    #&lt;br /&gt;
    # 1) configure syslog to accept network log events.  This is done&lt;br /&gt;
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in&lt;br /&gt;
    #    /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    # 2) configure local2 events to go to the /var/log/haproxy.log&lt;br /&gt;
    #   file. A line like the following can be added to&lt;br /&gt;
    #   /etc/sysconfig/syslog&lt;br /&gt;
    #&lt;br /&gt;
    #    local2.*                       /var/log/haproxy.log&lt;br /&gt;
    #&lt;br /&gt;
    log         127.0.0.1 local2&lt;br /&gt;
&lt;br /&gt;
    chroot      /var/lib/haproxy&lt;br /&gt;
    pidfile     /var/run/haproxy.pid&lt;br /&gt;
    maxconn     4000&lt;br /&gt;
    user        haproxy&lt;br /&gt;
    group       haproxy&lt;br /&gt;
    daemon&lt;br /&gt;
&lt;br /&gt;
    # turn on stats unix socket&lt;br /&gt;
    stats socket /var/lib/haproxy/stats&lt;br /&gt;
&lt;br /&gt;
    # utilize system-wide crypto-policies&lt;br /&gt;
    ssl-default-bind-ciphers PROFILE=SYSTEM&lt;br /&gt;
    ssl-default-server-ciphers PROFILE=SYSTEM&lt;br /&gt;
&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
# common defaults that all the 'listen' and 'backend' sections will&lt;br /&gt;
# use if not designated in their block&lt;br /&gt;
#---------------------------------------------------------------------&lt;br /&gt;
defaults&lt;br /&gt;
    log                     global&lt;br /&gt;
    option                  httplog&lt;br /&gt;
    option                  dontlognull&lt;br /&gt;
    option http-server-close&lt;br /&gt;
    option forwardfor       except 127.0.0.0/8&lt;br /&gt;
    option                  redispatch&lt;br /&gt;
    retries                 3&lt;br /&gt;
    timeout http-request    10s&lt;br /&gt;
    timeout queue           1m&lt;br /&gt;
    timeout connect         10s&lt;br /&gt;
    timeout client          30s&lt;br /&gt;
    timeout server          30s&lt;br /&gt;
    timeout http-keep-alive 10s&lt;br /&gt;
    timeout check           10s&lt;br /&gt;
    maxconn                 3000&lt;br /&gt;
&lt;br /&gt;
frontend kubernetes_api&lt;br /&gt;
    bind 172.17.100.100:6443&lt;br /&gt;
    default_backend kubernetes_api&lt;br /&gt;
&lt;br /&gt;
backend kubernetes_api&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:6443 check&lt;br /&gt;
    server master1 master1.openshift.stor:6443 check&lt;br /&gt;
    server master2 master2.openshift.stor:6443 check&lt;br /&gt;
    server master3 master3.openshift.stor:6443 check&lt;br /&gt;
&lt;br /&gt;
frontend machine_config&lt;br /&gt;
    bind 172.17.100.100:22623&lt;br /&gt;
    default_backend machine_config&lt;br /&gt;
&lt;br /&gt;
backend machine_config&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server bootstrap bootstrap.openshift.stor:22623 check&lt;br /&gt;
    server master1 master1.openshift.stor:22623 check&lt;br /&gt;
    server master2 master2.openshift.stor:22623 check&lt;br /&gt;
    server master3 master3.openshift.stor:22623 check&lt;br /&gt;
&lt;br /&gt;
frontend router_https&lt;br /&gt;
    bind 172.17.100.100:443&lt;br /&gt;
    default_backend router_https&lt;br /&gt;
&lt;br /&gt;
backend router_https&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    option ssl-hello-chk&lt;br /&gt;
    server worker1 worker1.openshift.stor:443 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:443 check&lt;br /&gt;
&lt;br /&gt;
frontend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    option httplog&lt;br /&gt;
    bind 172.17.100.100:80&lt;br /&gt;
    default_backend router_http&lt;br /&gt;
&lt;br /&gt;
backend router_http&lt;br /&gt;
    mode http&lt;br /&gt;
    balance roundrobin&lt;br /&gt;
    server worker1 worker1.openshift.stor:80 check&lt;br /&gt;
    server worker2 worker2.openshift.stor:80 check&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SElinux and firewall compatibility instructions are mandatory at this juncture:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;semanage  port -a -t http_port_t -p tcp 22623&lt;br /&gt;
semanage  port -a -t http_port_t -p tcp 6443&lt;br /&gt;
firewall-cmd --add-port=6443/tcp&lt;br /&gt;
firewall-cmd --add-port=22623/tcp&lt;br /&gt;
firewall-cmd --runtime-to-permanent&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Note, this assumes (correctly in the OACISS case) that the client.openshift.stor VM has only a single interface which it default places in the 'public' zone.&lt;br /&gt;
&lt;br /&gt;
= Setup process =&lt;br /&gt;
&lt;br /&gt;
== Ignition files ==&lt;br /&gt;
&lt;br /&gt;
After downloading &amp;lt;pre&amp;gt;openshift-client-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and &amp;lt;pre&amp;gt;openshift-install-linux-4.6.8.tar.gz&amp;lt;/pre&amp;gt; and unpacking them in /root/OCP/ on the client, it is time to generate the Ignition files that will automagically configure the virtual machines.&lt;br /&gt;
&lt;br /&gt;
These are created by openshift-install after reading a .yaml configuration file.&lt;br /&gt;
&lt;br /&gt;
This is the yaml given on the openshift install site as an example of a minimal configuration. Because we are installing the workers manually we must state 0 replicas for workers. Note that openshift-install helpfully deletes the input configuration yaml file, so this backup must be copied each time it is run...&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat install-config.yaml.bak &lt;br /&gt;
apiVersion: v1&lt;br /&gt;
baseDomain: stor&lt;br /&gt;
compute:&lt;br /&gt;
- hyperthreading: Enabled&lt;br /&gt;
  name: worker&lt;br /&gt;
  replicas: 0&lt;br /&gt;
controlPlane:&lt;br /&gt;
  hyperthreading: Enabled&lt;br /&gt;
  name: master&lt;br /&gt;
  replicas: 3&lt;br /&gt;
metadata:&lt;br /&gt;
  name: openshift&lt;br /&gt;
networking:&lt;br /&gt;
  clusterNetwork:&lt;br /&gt;
  - cidr: 10.128.0.0/14&lt;br /&gt;
    hostPrefix: 23&lt;br /&gt;
  networkType: OpenShiftSDN&lt;br /&gt;
  serviceNetwork:&lt;br /&gt;
  - 172.30.0.0/16&lt;br /&gt;
platform:&lt;br /&gt;
  none: {}&lt;br /&gt;
fips: false&lt;br /&gt;
pullSecret: '{}'&lt;br /&gt;
sshKey: 'ssh-ed25519 AAAA************ root@client.openshift.stor'&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Naturally, of course, the actual pullSecret '''and client SSH root trust key should be present.'''&lt;br /&gt;
&lt;br /&gt;
I have packaged the sequence of steps required next into a helpful prepare_install.sh script,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# cat prepare_install.sh p&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Deleting existing install logs and ign files&amp;quot;&lt;br /&gt;
rm -rf .openshift_install.log .openshift_install_state.json metadata.json bootstrap.ign worker.ign master.ign auth&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;cp install-config.yaml.bak install-config.yaml&amp;quot;&lt;br /&gt;
cp install-config.yaml.bak install-config.yaml&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating manifests&amp;quot;&lt;br /&gt;
./openshift-install create manifests --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;creating Ignition config files&amp;quot;&lt;br /&gt;
./openshift-install create ignition-configs --dir=./&lt;br /&gt;
&lt;br /&gt;
echo &amp;quot;Copying to web server on Pliny&amp;quot;&lt;br /&gt;
chmod go+r *ign&lt;br /&gt;
scp *ign root@pliny:/home/web_openshift/&lt;br /&gt;
&lt;br /&gt;
cp -f /root/OCP/auth/kubeconfig /root/.kube/config&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Install bootstrap node ==&lt;br /&gt;
&lt;br /&gt;
Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;vina# virt-install \&lt;br /&gt;
--name bootstrap \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/bootstrap.qcow2,size=20,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12:80/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
It will take something like 4 minutes for this to run and crank the bootstrap machine.&lt;br /&gt;
&lt;br /&gt;
Once this is done, ssh from client to core@bootstrap.openshift.stor and run 'journalctl | grep -i expired', and hopefully no output appears.&lt;br /&gt;
&lt;br /&gt;
If this succeeds, proceed.&lt;br /&gt;
&lt;br /&gt;
== Install master nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the bootstrap node is online, we can initiate installation of the master nodes,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
&lt;br /&gt;
virt-install \&lt;br /&gt;
--name master$NODE \&lt;br /&gt;
--vcpus 8 \&lt;br /&gt;
--ram 16384 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/master$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The $NODE substitution increments the virtual disk name, IP and hostname appropriately. We, here, have just the three master nodes as 100.101, 2 and 3.&lt;br /&gt;
&lt;br /&gt;
These should take 5-ish minutes to install, boot, self-update and reboot before they try and contact the hive mind.&lt;br /&gt;
&lt;br /&gt;
Several problems can manifest at this point, all caused by misconfiguration on the load balancer,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   88.232297] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&lt;br /&gt;
[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22&lt;br /&gt;
[   93.245517] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: http: server gave HTTP response to HTTPS client&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The problem lies in the haproxy configuration file. Under 'global' do not have 'mode http'. This is given correctly in the haproxy config above.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[   ***] A start job is running for Ignition (fetch) (31min 30s / no limit)[ 1893.903489] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #381&lt;br /&gt;
[ 1893.921039] ignition[698]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: x509: certificate signed by unknown authority&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This occurred when, in initially configuring haproxy, I accidentally told the forwarder for port 22623 to talk to master*:6443.&lt;br /&gt;
&lt;br /&gt;
Another possible problem is&lt;br /&gt;
&amp;lt;pre&amp;gt;[***   ] A start job is running for Ignition (fetch) (1min 45s / no limit)[  108.236817] ignition[690]: GET https://api-int.openshift.stor:22623/config/master: attempt #25&lt;br /&gt;
[  108.249306] ignition[690]: GET error: Get &amp;quot;https://api-int.openshift.stor:22623/config/master&amp;quot;: dial tcp 172.17.100.100:22623: connect: no route to host&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This error is provoked for several possible reasons relating to the load balancer setup (See load balancer section). Most likely either&lt;br /&gt;
* Balancer not running [TCP/SYN rejected]&lt;br /&gt;
* Firewall misconfigured [TCP/SYN being dropped]&lt;br /&gt;
&lt;br /&gt;
== Install completion and bootstrap shutdown ==&lt;br /&gt;
&lt;br /&gt;
From the OCP directory on the client, once the master nodes launch into their self-setup process, run&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;./openshift-install --dir=./ wait-for bootstrap-complete --log-level=info&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This command will block your terminal window until the master nodes finish setting themselves up (a 10+ minute endeavour in my case), thereupon informing you it is safe to shut the bootstrap machine down; &amp;lt;pre&amp;gt;virsh shutdown bootstrap&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, running &amp;lt;pre&amp;gt;[root@client OCP]# ./oc get co&amp;lt;/pre&amp;gt; should vomit out 30 or so lines of the form &amp;quot;NAME [same version] True ...&amp;quot;.&lt;br /&gt;
&lt;br /&gt;
If we get nodes, we now (technically) have a working cluster,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Worker install ==&lt;br /&gt;
&lt;br /&gt;
Now it is time to install the worker nodes. This may be done on the same machine, or across whatever real machines are going to run the cluster,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
virt-install \&lt;br /&gt;
--name worker$NODE \&lt;br /&gt;
--vcpus 32 \&lt;br /&gt;
--ram 65536 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/worker$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Like the install commands for master nodes, the commands for the workers are clones, save for incrementing the IP addresses, workerN hostnames and the virtual disk names.&lt;br /&gt;
&lt;br /&gt;
== Worker install on IBM s924 ==&lt;br /&gt;
&lt;br /&gt;
First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).&lt;br /&gt;
&lt;br /&gt;
Then we must ssh to the VIOS and&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup_env&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
scp erik-k@172.17.202.18:~/downloads/rhcos-4.6.8-ppc64le-live.ppc64le.iso ./&amp;lt;/pre&amp;gt;&lt;br /&gt;
to copy the rhcos installer image to the vios, then use the system -&amp;gt; virtual storage -&amp;gt; vios -&amp;gt; manage -&amp;gt; virtual optical media to add it and assign to the rhcos LPAR.&lt;br /&gt;
&lt;br /&gt;
Once this is done and we have verified that the lpar will boot &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;NODE=1&lt;br /&gt;
virt-install \&lt;br /&gt;
--name worker$NODE \&lt;br /&gt;
--vcpus 32 \&lt;br /&gt;
--ram 65536 \&lt;br /&gt;
--disk path=/var/lib/libvirt/images/worker$NODE.qcow2,size=32,format=qcow2,bus=virtio \&lt;br /&gt;
--graphics none \&lt;br /&gt;
--serial pty \&lt;br /&gt;
--console=pty,target_type=virtio \&lt;br /&gt;
--network network=ocp,model=virtio \&lt;br /&gt;
--extra-args &amp;quot;ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign &amp;quot; \&lt;br /&gt;
--os-type linux --os-variant rhel7.0 \&lt;br /&gt;
--location http://172.17.202.12/openshift_ppc64le/&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Approve new nodes ==&lt;br /&gt;
&lt;br /&gt;
Once the consoles for the worker nodes are sitting at the login prompt, we can add them to the cluster.&lt;br /&gt;
&lt;br /&gt;
Running &amp;lt;pre&amp;gt;./oc get csr&amp;lt;/pre&amp;gt; will show that we have two key requests waiting from the workers.&lt;br /&gt;
&lt;br /&gt;
Do &amp;lt;pre&amp;gt;./oc adm certificate approve $NAME&amp;lt;/pre&amp;gt; for each of the two NAMEd requests to inject the nanoprobes and make them part of the collective.&lt;br /&gt;
&lt;br /&gt;
After about 10-15 seconds we can get nodes again and see they have appeared,&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES           AGE    VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker          109s   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker          102s   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
We note that the master nodes are schedulable because we originally set ourselves up with no worker nodes. Running&lt;br /&gt;
&amp;lt;pre&amp;gt;./oc edit schedulers.config.openshift.io cluster&amp;lt;/pre&amp;gt;&lt;br /&gt;
And edit the line near the bottom for master schedulable from 'true' to 'false'. Now we have what we want:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;[root@client OCP]# ./oc get nodes &lt;br /&gt;
NAME                     STATUS   ROLES    AGE   VERSION&lt;br /&gt;
master1.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master2.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
master3.openshift.stor   Ready    master   44m   v1.19.0+7070803&lt;br /&gt;
worker1.openshift.stor   Ready    worker   10m   v1.19.0+7070803&lt;br /&gt;
worker2.openshift.stor   Ready    worker   10m   v1.19.0+7070803&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Post-install smart moves ==&lt;br /&gt;
&lt;br /&gt;
Now that it's working, another very minor not at all noteworthy thing to mention...&lt;br /&gt;
&lt;br /&gt;
If the system ever goes down for more than 24 hr, it will be impossible to restart. So, now that it's working, this would be a great time to go run 'virsh list' and then 'virsh autostart X' all the domains: client, master[1 | 2 | 3] and worker[1 | 2] to make sure they come back up even if the host restarts.&lt;br /&gt;
&lt;br /&gt;
Now sit down and pour yourself a nice scotch, you deserve it.&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3322</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3322"/>
		<updated>2021-12-21T21:29:02Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
&lt;br /&gt;
The [[Service:storage]] describes available storage for users of OACISS systems.&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
OACISS has a large amount of storage available: See [[Service:storage | Storage]].&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Deschutes machine room&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node + Intel DG1|| RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 10GbE || R84.U11&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || 1GbE || R85.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3321</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3321"/>
		<updated>2021-12-20T18:02:19Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Nimbus is online.&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
&lt;br /&gt;
The [[Service:storage]] describes available storage for users of OACISS systems.&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
OACISS has a large amount of storage available: See [[Service:storage | Storage]].&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Deschutes machine room&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node + Intel DG1|| RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || ? || ?&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || ? || ?&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || ? || ?&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3320</id>
		<title>Category:Servers</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Category:Servers&amp;diff=3320"/>
		<updated>2021-12-01T23:16:52Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: Cumulus-AIX.stor&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This is a list of all OACISS servers in the Franken-cluster. You may also select a section header to view the Wiki-generated category index for systems of that type.&lt;br /&gt;
&lt;br /&gt;
[[File:frankenstein.png|128px]]&lt;br /&gt;
&lt;br /&gt;
The [[NetworkInfrastructure]] page describes the host naming (dns) conventions, as well as documenting the physical setup and connections within the OACISS racks in the machine room. All OACISS systems automatically search .nic.uoregon.edu for DNS, so only the short hostname is needed for ssh internally.&lt;br /&gt;
&lt;br /&gt;
The [[Service:storage]] describes available storage for users of OACISS systems.&lt;br /&gt;
&lt;br /&gt;
Click on the server links to access more information about individual machines. Note that only the two machines designated as login gateways (orthus, cerberus) are accessible by machines outside of nic.uoregon.edu.&lt;br /&gt;
&lt;br /&gt;
OACISS has a large amount of storage available: See [[Service:storage | Storage]].&lt;br /&gt;
&lt;br /&gt;
{| border=&amp;quot;1&amp;quot; cellspacing=&amp;quot;0&amp;quot; cellpadding=&amp;quot;5&amp;quot;&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Deschutes machine room&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processors !! Local Network !! Physical location&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Orthus]] || '''Primary login gateway''' || Rhel-8.4 || Dell PowerEdge || 2 x 8c Xeon E5-2667 v2 @ 3.3GHz || 10GbE || &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jupiter]] || Quad Cooper lake + Intel DG1 || Ubuntu 20.04.2 || Supermicro Sys-240 || 4 x 24c Xeon Gold 6438 @ 2.3GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Saturn]] || Quad Cooper lake + A100 (80GB) || Ubuntu 20.04.2 || Gigabyte RS292-4S1 || 4 x 26c Xeon Platinum 8367HC @ 3.2GHz || 100GbE + EDR || R86.U10&lt;br /&gt;
|- &lt;br /&gt;
| [[Compute: Reptar]] || Cascade lake 6248 node + Intel DG1|| RHEL 8.4 || Supermicro 7049 || 2 x 24c Xeon Gold 6248R @ 2.9GHz || 10GbE + 100GbE || R84.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Illyad]] || AMD + 2 A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Rome 7402 @ 2.8GHz || 100GbE + 2xEDR || R85.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gilgamesh]] || AMD + 2 MI50 + A100 (40GB) || Centos 8.3 || Preproduction SuperMicro || 2 x 24c Epyc Milan 7413 @ 2.6GHz || 100GbE + 2xEDR || R85.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Instinct]] || Intel + 2 AMD MI100 + MI50 || Centos 7.9 || Supermicro SC747 || 2 x 14c Xeon E5-2660 v4 2.0GHz || 100GbE || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Voltar]] || A100 (80GB) + P100 + V100 GPU node || Centos 7.8 || Cascade Lake GPU server || 2 x 16c Xeon Gold 6226R @ 2.9GHz || 10GbE + EDR || R86.U26&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cyclops]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Gorgon]] || IBM Power9 + 4 V100 || RHEL 7.6 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE + 2xHDR (200 Gbps)|| R86.U16&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Medusa]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U14&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Typhon]] || IBM Power9 || RHEL 8.4 || IBM AC922 || 2 x 20c Power9 @ 3.66GHz || 10GbE || R86.U12&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Delphi]] || Intel + GV100 || Centos-7.8 || Intel SDP || 2 x 18c Xeon E5-2697 v4 || 100GbE || R86.U35&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Aurora]] || NEC SX-Aurora demo machine || Centos 7.9 || 2 x NEC SX-Aurora Tsubasa Vector Engine || 8c Xeon 4108 Silver @ 1.8GHz || 10GbE + EDR || R85.U31&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Godzilla]] || Intel + 2 x K80 node || RHEL 8.2 || Broadwell GPU server || 2 x 14c Xeon E5-2680v4 @ 2.3GHz || 40GbE + EDR || R85.U6&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Centaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U18&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Minotaur]] || IBM Power8 + 2 K80 || Ubuntu 20.04 || IBM S822LC || 2 x 20c Power8 @ 3.5GHz || 10GbE || R85.U20&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Eagle]] || IBM Power9 + 3 x T4 || Ubuntu 20.04 || IBM IC922 || 2 x 16c Power9 @ 2.1GHz || 10GbE + 2xEDR || R86.U24&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pegasus]] || Compute node || Centos 7.8 || Intel Skylake server || 2 x 18c Xeon Gold 6140 @ 2.3GHz || 100GbE + EDR || R86.U22&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Vina]] || Raptor Talos II || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U44&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Pike]] || Raptor Talos II + MI25 || Ubuntu-20 || Talos workstation || 2 x 22c Power9 @ 2.2GHz || 10GbE || R84.U29&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cirrus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || ? || ?&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Cumulus-AIX.stor]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || ? || ?&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Nimbus]] || AIX machine || AIX 7.2 || IBM S-924 server || 2 x 20c Power9 @ 3.6GHz || ? || ?&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: KNL Grover]] || Intel Phi system || Centos 7.8 || Intel KNL server || 68c Xeon Phi 7250 @ 1.4GHz || 1GbE || R86.U20&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Compute|Compute Nodes]] in Streisinger&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! OS !! Model !! Processor !! Local Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Chymera]] || Drives 8K display in 472 || Centos 7 || Dell T620 || 2 x 10c Xeon E5-2680 v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Visualization: Cerberus]] || '''Secondary login gateway'''; Jetson/Nucs + NFS ||  Centos-7 || Dell T620|| 2 x 10c Xeon E5-2680v2 @ 2.8GHz || 10GbE || Str-470 window&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: NUC cluster|NUC cluster]] || Intel NUCs (16) || Centos 8.2 || 16 x NUC 4250 || 4c Intel i5-4250 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-1 || Ubuntu-18.04.3 || 12 x Jetson-TX1 || 4c ARM V8l rev 1 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Jetson cluster|Jetson ARM64 cluster]] || Tegra TX-2|| Ubuntu-16.04.05 || 4 x Jetson-TX2 || 4c ARM V8l rev 3 || 1GbE || Str 470 mini-rack&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Xavier]] || NVidia Tegra 3 || Ubuntu-18.04.3 || Jetson TX-3 || 8c ARM v8l rev 0 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: OD1K]] || ARM64 v8 || Ubuntu || SoftIron || ARM64 || 1GbE || Str-470 mini-rack adjacent&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Omicron]] || M1 Mac || OSX || M1 Mini || M1 || 1GbE || Str-470 foyer&lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Sever]] || Intel Xe || Ubuntu 20 || XPS 13 || Quad core i7 Gen11 @ 2.8GHz || 10GbE || Str-470 foyer &lt;br /&gt;
|-&lt;br /&gt;
| [[Compute: Silicon]] || VLSI simulation node || Debian 10 || Supermicro mobo || 6c 3.6GHz Broadwell CPU || 1GbE || Str-473&lt;br /&gt;
|-&lt;br /&gt;
! colspan=7 align=center | [[:Category:Infrastructure|Infrastructure Nodes]]&lt;br /&gt;
|-&lt;br /&gt;
! Name !! Description !! Model !! Processor !! Network !! Physical location &lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:orion]] || VM host || SuperMicro || 16c Xeon Platinum || 10GbE || R35.U37&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mecha]] || ? || Silicon Mechanics || 2x Xeon E5410 || 1GbE || R34.U37 left&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:newstorage]] || NFS Server || Silicon Mechanics || 4c Xeon E5620 || 2x1GbE || R34.U9&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:mnemosyne]] || NFS Server || Silicon Mechanics || 8c Xeon Silver 4112 || 40GbE + EDR || R35.21&lt;br /&gt;
|-&lt;br /&gt;
| [[Infrastructure:lighthouse]] || Backup infrastructure || Qlogic Comet HA600 || Core i5-10500 x6 @ 2.3GHz || 1GbE || Str-470&lt;br /&gt;
&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[ComputeSkeleton]] - Outline for new machine entries&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3319</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3319"/>
		<updated>2021-11-23T20:59:50Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Yum */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo vim tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz 0mq-devel binutils-devel coreutils blas-devel openblas-devel matplotlib libjpeg-devel openblas-devel ImageMagick-devel vim-X11 lua-devel tcl-devel tk-devel findutils gdb cmake &amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
Edit /root/.bashrc:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;PATH=&amp;quot;/opt/freeware/bin:$PATH&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
=== Modules setup ===&lt;br /&gt;
&lt;br /&gt;
The /packages directory is already mounted by the nfs setup step, but environment modules are not setup.&lt;br /&gt;
&lt;br /&gt;
In /etc/profile:&lt;br /&gt;
&amp;lt;pre&amp;gt;export PATH=/opt/freeware/bin/:$PATH&lt;br /&gt;
export PATH=/opt/IBM/openxlf/default/bin:$PATH&lt;br /&gt;
export PATH=/opt/IBM/xlC/default/bin:$PATH&lt;br /&gt;
source /usr/local/packages/modules-5.0.1/init/bash&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will avoid the need for everyone to edit their bashrc&lt;br /&gt;
&lt;br /&gt;
=== IBM xL compilers ===&lt;br /&gt;
&lt;br /&gt;
=== Python ===&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3318</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3318"/>
		<updated>2021-11-23T20:48:27Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo vim tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz seromq-devel binutils-devel coreutils blas-devel openblas-devel matplotlib libjpeg-devel openblas-devel ImageMagick-devel vim-X11 lua-devel tcl-devel tk-devel findutils gdb&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
Edit /root/.bashrc:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;PATH=&amp;quot;/opt/freeware/bin:$PATH&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
=== Modules setup ===&lt;br /&gt;
&lt;br /&gt;
The /packages directory is already mounted by the nfs setup step, but environment modules are not setup.&lt;br /&gt;
&lt;br /&gt;
In /etc/profile:&lt;br /&gt;
&amp;lt;pre&amp;gt;export PATH=/opt/freeware/bin/:$PATH&lt;br /&gt;
export PATH=/opt/IBM/openxlf/default/bin:$PATH&lt;br /&gt;
export PATH=/opt/IBM/xlC/default/bin:$PATH&lt;br /&gt;
source /usr/local/packages/modules-5.0.1/init/bash&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will avoid the need for everyone to edit their bashrc&lt;br /&gt;
&lt;br /&gt;
=== IBM xL compilers ===&lt;br /&gt;
&lt;br /&gt;
=== Python ===&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3317</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3317"/>
		<updated>2021-11-23T20:14:02Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo vim tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz seromq-devel binutils-devel coreutils blas-devel openblas-devel matplotlib libjpeg-devel openblas-devel ImageMagick-devel vim-X11 lua-devel tcl-devel tk-devel findutils gdb&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
Edit /root/.bashrc:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;PATH=&amp;quot;/opt/freeware/bin:$PATH&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
=== Modules setup ===&lt;br /&gt;
&lt;br /&gt;
The /packages directory is already mounted by the nfs setup step, but environment modules are not setup.&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3316</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3316"/>
		<updated>2021-11-23T20:08:26Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Yum */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo vim tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz seromq-devel binutils-devel coreutils blas-devel openblas-devel matplotlib libjpeg-devel openblas-devel ImageMagick-devel vim-X11 lua-devel tcl-devel tk-devel findutils gdb&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
Edit /root/.bashrc:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;PATH=&amp;quot;/opt/freeware/bin:$PATH&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
.......&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3315</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3315"/>
		<updated>2021-11-23T20:05:50Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Yum */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo vim tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz seromq-devel binutils-devel coreutils blas-devel openblas-devel matplotlib libjpeg-devel openblas-devel&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
Edit /root/.bashrc:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;PATH=&amp;quot;/opt/freeware/bin:$PATH&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
.......&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3314</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3314"/>
		<updated>2021-11-23T20:05:06Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Yum */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo vim tar gcc-gfortran gcc-c++ emacs blas freetype2-devel lapack libpng-devel python3-devel xz seromq-devel binutils-devel coreutils blas-devel openblas-devel matplotlib libjpeg-devel openblas-devel&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
.......&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3313</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3313"/>
		<updated>2021-11-23T19:38:46Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Yum */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo vim tar&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
.......&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
	<entry>
		<id>https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3312</id>
		<title>HMC AIX Setup</title>
		<link rel="alternate" type="text/html" href="https://systems.nic.uoregon.edu/internal-wiki/index.php?title=HMC_AIX_Setup&amp;diff=3312"/>
		<updated>2021-11-23T19:36:54Z</updated>

		<summary type="html">&lt;p&gt;Nic-wiki: /* Sudo setup */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page will document the installation procedure for a system controlled by an IBM (v)HMC.&lt;br /&gt;
&lt;br /&gt;
The installation steps are in summary,&lt;br /&gt;
* Install VIOS&lt;br /&gt;
* Create virtual network bridges&lt;br /&gt;
* Create virtual disks and assign&lt;br /&gt;
* Install AIX&lt;br /&gt;
* Setup AIX&lt;br /&gt;
&lt;br /&gt;
Hardware prerequisites assumed:&lt;br /&gt;
- At least 1 storage device on SATA/SAS&lt;br /&gt;
- Network port connected to HMC&lt;br /&gt;
- At least 2 external network ports connected, 1 to public &amp;amp; 1 to private network&lt;br /&gt;
&lt;br /&gt;
== VIOS install ==&lt;br /&gt;
&lt;br /&gt;
The VIOS install uses a VIOS image stored on the HMC to bootstrap the server. Once the HMC is installed, it will be necessary to SSH to it, and use&lt;br /&gt;
&amp;lt;pre&amp;gt;chfs -a size=+10G /&amp;lt;/pre&amp;gt;&lt;br /&gt;
to grow the default filesystem, create a directory on the HMC, and scp the vios-* image to there. The current (3.1) VIOS image is stored in ~erik-k's downloads as&lt;br /&gt;
&amp;lt;pre&amp;gt;PwrVMVIOSBIV3.1.3.10Fls92021.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
References:&lt;br /&gt;
&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=hmc-installing-vios&lt;br /&gt;
&lt;br /&gt;
Top -&amp;gt; System -&amp;gt; 'create vio server'&lt;br /&gt;
&lt;br /&gt;
Clicking through menus, assign 2 processors, the NICs and at least one SAS adapter with disk (vios will not install on nvme).&lt;br /&gt;
&lt;br /&gt;
Select 'management console images' to install and 'vios-3.1.3.10-flash'. If an image isn't present, see top of this section to upload it.&lt;br /&gt;
&lt;br /&gt;
It is necessary to assign a working ethernet port to the vios. All 'm c image install' does is just temporarily throw up a NIM server, install and then tear it down. Make certain that the assigned network port is on the correct switch &amp;amp; that the switch has that port on the right vlan.&lt;br /&gt;
&lt;br /&gt;
I have decided to use 172.17.20.x for the VIOS ethernets. Enter 172.17.202.79 (vina IP) for gateway... this does not work but it needs something.&lt;br /&gt;
&lt;br /&gt;
Click install, and go have lunch. I've clocked this process in at around 45 minutes. It may be that temporarily assigning more processors would speed it up?&lt;br /&gt;
&lt;br /&gt;
The finishing of the install procedure is unreliable. It may or may not report &amp;quot;done&amp;quot;. As long as it does _not_ report failure, once it is done, you can try to click 'accept license.'&lt;br /&gt;
&lt;br /&gt;
Next step:&lt;br /&gt;
&lt;br /&gt;
Server -&amp;gt; vio servers -&amp;gt; [click server] -&amp;gt; vios actions -&amp;gt; console -&amp;gt; open terminal&lt;br /&gt;
&lt;br /&gt;
After a bit, a godawfully barely-functional console will ask to run.&lt;br /&gt;
&lt;br /&gt;
Enter a password&lt;br /&gt;
&lt;br /&gt;
Accept license.&lt;br /&gt;
&lt;br /&gt;
Run oem_setup_env to get prompted again, and accept license.&lt;br /&gt;
&lt;br /&gt;
'''Okay, new blank slate VIOS is installed.'''&lt;br /&gt;
&lt;br /&gt;
=== Partitions ===&lt;br /&gt;
&lt;br /&gt;
At this point we also want to create OS partitions because we will need them available when we setup virtual disks&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; top 'create partition' -&amp;gt; name it&lt;br /&gt;
&lt;br /&gt;
== VIOS configuration ==&lt;br /&gt;
&lt;br /&gt;
Virtual networking configuration:&lt;br /&gt;
https://www.ibm.com/docs/en/power9?topic=avnw-adding-virtual-network-by-creating-virtual-network-bridge&lt;br /&gt;
&lt;br /&gt;
System -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpriv&lt;br /&gt;
Bridged: yes, tagged no, pvid 172, use default switch, next&lt;br /&gt;
&lt;br /&gt;
Enable jumbo frames [critical for private network!] and LSO, next&lt;br /&gt;
&lt;br /&gt;
Assign backing device from list. '''Note: This must be the correct device. The virbr setup process rewrites the MTU for the backing device when jumbo frames are enabled, and you *can't* reset it from inside the vios easily it seems.''' If this is chosen wrong, the only apparent option is delete the whole virbr and start over.&lt;br /&gt;
&lt;br /&gt;
It will be created and use the default 802.3 virtual switch.&lt;br /&gt;
&lt;br /&gt;
Now go back and greate the public bridge:&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; powervm -&amp;gt; virtual networks -&amp;gt; add virtual network&lt;br /&gt;
&lt;br /&gt;
name: brpub&lt;br /&gt;
Bridged: yes, tagged no, pvid 128, check 'advanced' and use new virtual switch, next&lt;br /&gt;
&lt;br /&gt;
ok, create&lt;br /&gt;
&lt;br /&gt;
=== Virtual install library setup ===&lt;br /&gt;
&lt;br /&gt;
First we need to import OS install images to the VIOS (much like we used the hmc to bootstrap the vios, the vios needs the image to bootstrap the partition).&lt;br /&gt;
&lt;br /&gt;
SSH to the vios, username 'padmin'&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;oem_setup-env&lt;br /&gt;
mkdir -p /Maingroup/images&lt;br /&gt;
cd /Maingroup/images&lt;br /&gt;
chfs -a size=+15G / # to enlarge storage sufficiently&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
using scp, ISO images are available on erik-k's downloads:&lt;br /&gt;
&amp;lt;pre&amp;gt;aix_7200-05-03-2136_flash_092021.iso&lt;br /&gt;
rhel-8.2-ppc64le-dvd.iso&lt;br /&gt;
ubuntu-20.04.1-live-server-ppc64el.iso&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
SCP these to to /Maingroup/images&lt;br /&gt;
&lt;br /&gt;
Now go to system -&amp;gt; virtual storage -&amp;gt; vio server -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; create virtual library -&amp;gt; 25GB&lt;br /&gt;
&lt;br /&gt;
optical devices -&amp;gt; action -&amp;gt; add media -&amp;gt; from existing file&lt;br /&gt;
&lt;br /&gt;
/Maingrouop/images/ubuntu-20.04.1-liver-server-ppc64el.iso e.g.&lt;br /&gt;
&lt;br /&gt;
Quite insanely you have to type the entire filename manually, there is no browser box. *blink blink*.&lt;br /&gt;
&lt;br /&gt;
=== Partition configuration ===&lt;br /&gt;
&lt;br /&gt;
systems -&amp;gt; powervm -&amp;gt; virtual storage&lt;br /&gt;
&lt;br /&gt;
select vio server -&amp;gt; action -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
click storage pools -&amp;gt; Create a storage pool &amp;amp; assign the NVMEs to it&lt;br /&gt;
&lt;br /&gt;
click virtual disks -&amp;gt; create&lt;br /&gt;
&lt;br /&gt;
name: 'aixroot' or something&lt;br /&gt;
pool: nvmepool&lt;br /&gt;
size: 500G e.g.&lt;br /&gt;
assign to partition: aix-part &lt;br /&gt;
&lt;br /&gt;
create &amp;amp; assign adapter on partition -&amp;gt; yes please&lt;br /&gt;
&lt;br /&gt;
Now click system -&amp;gt; partitions -&amp;gt; [partition] -&amp;gt; virtual networks -&amp;gt; attach virtual network&lt;br /&gt;
&lt;br /&gt;
[X] show and attaach new adapters&lt;br /&gt;
[X] check brpub and brpriv to connect to both networks&lt;br /&gt;
&lt;br /&gt;
Click system -&amp;gt; virtual storage -&amp;gt; [vios] -&amp;gt; manage&lt;br /&gt;
&lt;br /&gt;
virtual optical devices -&amp;gt; [select image] -&amp;gt; modify assignment to partition we're installing -&amp;gt; ok&lt;br /&gt;
&lt;br /&gt;
== AIX install ==&lt;br /&gt;
&lt;br /&gt;
After assigning virtual networks, virtual disk drive &amp;amp; aix-7.2 install media,&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; start&lt;br /&gt;
&lt;br /&gt;
system -&amp;gt; partitions -&amp;gt; aix partition -&amp;gt; system actions -&amp;gt; console -&amp;gt; open&lt;br /&gt;
&lt;br /&gt;
'''AIX INSTALL CRITICAL''': Select software options, and install both openssh client and server, or you'll be kicking yourself in the balls to distract yourself from the pain while you try to find another way to install it after.&lt;br /&gt;
&lt;br /&gt;
Look at the devices and confirm you have a scsi disk and a cd drive&lt;br /&gt;
&lt;br /&gt;
== AIX configuration ==&lt;br /&gt;
&lt;br /&gt;
Upon firstboot, smitty comes up.&lt;br /&gt;
&lt;br /&gt;
Set timezone to use/los angeles&lt;br /&gt;
Use passwd to set password&lt;br /&gt;
&lt;br /&gt;
Run &amp;lt;pre&amp;gt;df -sm&amp;lt;/pre&amp;gt; to see what an infinitesimal size of disk has been assigned to a bunch of partitions.&lt;br /&gt;
&lt;br /&gt;
Use &amp;lt;pre&amp;gt;chfs -a size=+xG /filesystem&amp;lt;/pre&amp;gt; to grow them. Remember we have 500G of virtual disk!&lt;br /&gt;
&lt;br /&gt;
mkdir /root&lt;br /&gt;
&lt;br /&gt;
cd /etc, open passwd in vi,&lt;br /&gt;
&lt;br /&gt;
edit root's homedir to /root&lt;br /&gt;
&lt;br /&gt;
=== Ethernet configuration ===&lt;br /&gt;
&lt;br /&gt;
lsdev | grep -i eth -&amp;gt; should print en0 and en1&lt;br /&gt;
&lt;br /&gt;
cd /etc&lt;br /&gt;
vi dhcpcd.ini&lt;br /&gt;
&lt;br /&gt;
Page down to the bottom, append,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;interface en0 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&lt;br /&gt;
interface en1 {&lt;br /&gt;
  option 12 &amp;quot;cirrus-aix&amp;quot;&lt;br /&gt;
}&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
but of course use the right hostname. Be very very careful in VI, the hmc terminal is broken! cat the file out after to be sure it did it right.&lt;br /&gt;
&lt;br /&gt;
Use system -&amp;gt; virtual networking diagram to find which interface is connected to which network.&lt;br /&gt;
&lt;br /&gt;
Use ifconfig -a to get MAC addresses&lt;br /&gt;
&lt;br /&gt;
Edit named on cato and dhcpd on mnemosyne to make sure DNS and IP assignment work correctly.&lt;br /&gt;
&lt;br /&gt;
Once this is done,&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s dhcpcd&amp;lt;/pre&amp;gt;&lt;br /&gt;
to connect and get IP addresses.&lt;br /&gt;
&lt;br /&gt;
Now we have to configure jumbo frames on the host,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;ifconfig -a&amp;lt;/pre&amp;gt; will reveal which adapter is mated to the private network, N&lt;br /&gt;
&lt;br /&gt;
See:&lt;br /&gt;
https://developer.ibm.com/articles/au-aix-largesend-jumboframes/&lt;br /&gt;
&lt;br /&gt;
If N is 0, this can be pasted - as one line - to restart the sucker and set jumbo frames on,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chdev -l en0 -a state=down; chdev -l en0 -a state=detach; chdev -l ent0 -a jumbo_frames=yes; chdev -l en0 -a mtu=9000; chdev -l en0 -a state=up; mkdev -l inet0&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check &amp;lt;pre&amp;gt;lsattr -El (device) | grep -e mtu -e jumbo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Check afterwards from another oaciss machine (because aix ping command is stupid) too:&lt;br /&gt;
&amp;lt;pre&amp;gt;orthus# ping -s 9000 cumulus-aix.stor&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== SSH===&lt;br /&gt;
&lt;br /&gt;
Go to /etc/ssh and edit sshd_config&lt;br /&gt;
&lt;br /&gt;
Change ListenAddress to the private interface address for now.&lt;br /&gt;
&lt;br /&gt;
stopsrc -s sshd&lt;br /&gt;
startsrc -s sshd&lt;br /&gt;
&lt;br /&gt;
This will temporarily secure the system by restricting ssh to the private network&lt;br /&gt;
&lt;br /&gt;
=== Care and feeding package ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;scp erik-k@orthus:~/ibmset.tar /root&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This will fetch the GSkit, ldap client, ldap licence and yum installers to the system all in one swoop. Untar it, it is not a tarbomb.&lt;br /&gt;
&lt;br /&gt;
=== Yum === &lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd yum; rpm -ivh *&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Once this is installed,&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;yum install -y bash wget sudo&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Bash is now available, yay&lt;br /&gt;
&lt;br /&gt;
=== LDAP ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset&lt;br /&gt;
uncompress 20151204_GSKit8_8_0_50_44.tar.Z&lt;br /&gt;
tar -xf 20151204_GSKit8_8_0_50_44.tar&lt;br /&gt;
cd 20151204_GSKit8_8_0_50_44&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
use smitty to install '''ALL FOUR packages''', not just the 64 bit ones.&lt;br /&gt;
&lt;br /&gt;
The following are the confirmed working installations on Cirrus:&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1# lslpp -L | grep -i gskit&lt;br /&gt;
  GSKit8.gskcrypt32.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskcrypt64.ppc.rte&lt;br /&gt;
                           8.0.50.44    C     F    IBM GSKit Cryptography Runtime&lt;br /&gt;
  GSKit8.gskssl32.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  GSKit8.gskssl64.ppc.rte  8.0.50.44    C     F    IBM GSKit SSL Runtime With&lt;br /&gt;
  gpfs.gskit               8.0.55.19    C     F    GPFS GSKit Cryptography&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
'''Utilize the following instructions ONLY if all aix systems are gone and a new install is needed:'''&lt;br /&gt;
&lt;br /&gt;
https://www.unix.com/aix/261855-aix-ldap-client-authenticate-against-linux-openldap-server-over-tls-ssl.html&lt;br /&gt;
[root@cato openldap]# openssl pkcs12 -export -in /etc/openldap/certs/newslapd_cert.pem -inkey /etc/openldap/certs/newslapd_key_nocrypt.pem -out newslapd.p12 -name &amp;quot;CA Signed&amp;quot;&lt;br /&gt;
(enter 'Password' for password on key)&lt;br /&gt;
&lt;br /&gt;
bash-5.1# gsk8capicmd_64 -cert -import -db /root/newslapd.p12 -pw Password -target /etc/security/ldap/key.kdb&lt;br /&gt;
&lt;br /&gt;
'''End special instructions'''&lt;br /&gt;
&lt;br /&gt;
Normal instructions: copy /etc/security/ldap/key.kdb from another working aix system.&lt;br /&gt;
&lt;br /&gt;
Next per&lt;br /&gt;
https://www.ibm.com/docs/en/aix/7.2?topic=module-setting-up-ldap-client&lt;br /&gt;
&lt;br /&gt;
we need to run idsLicense.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;cd /root/ibmset/license&lt;br /&gt;
tar -xf idslic.tar&lt;br /&gt;
./idsLicense&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now cd /root/ibmset and use smitty to install all idsldap files. Once it is successful we should be able to see&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;bash-5.1#  lslpp -L | grep -i idsl&lt;br /&gt;
  idsldap.clt32bit64.rte    6.4.0.23    C     F    Directory Server - 32 bit&lt;br /&gt;
  idsldap.clt64bit64.rte    6.4.0.23    C     F    Directory Server - 64 bit&lt;br /&gt;
  idsldap.clt_max_crypto32bit64.rte&lt;br /&gt;
  idsldap.clt_max_crypto64bit64.rte&lt;br /&gt;
  idsldap.cltbase64.adt     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.cltbase64.rte     6.4.0.23    C     F    Directory Server - Base Client&lt;br /&gt;
  idsldap.license64.rte     6.4.0.23    C     F    Directory Server - License&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
bash-5.1# # mksecldap -c -a 'cn=anonymous,dc=nic,dc=uoregon,dc=edu' -p 'actualpasswordhere' -A ldap_auth -S rfc2307 -d 'dc=nic,dc=uoregon,dc=edu' -h ldap1.nic.uoregon.edu,ldap2.nic.uoregon.edu -k /etc/security/ldap/key.kdb -w Password -u SYSTEM&lt;br /&gt;
&lt;br /&gt;
Checking:&lt;br /&gt;
https://www.ibm.com/support/pages/active-directory-ad-aix-step-step-instructions-integrate-active-directory-2016-aix-ldap-protocol&lt;br /&gt;
&lt;br /&gt;
# lsuser -f -a id pgrp groups home shell SYSTEM registry erik-k&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;erik-k:&lt;br /&gt;
        id=15382&lt;br /&gt;
        pgrp=nic&lt;br /&gt;
        groups=nic,nicadmin,paraducks,webadmin,webuser,swmgr,lsfadmin&lt;br /&gt;
        home=/home/users/erik-k&lt;br /&gt;
        shell=/bin/bash&lt;br /&gt;
        SYSTEM=compat&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
and&lt;br /&gt;
#lsgroup -f nic&lt;br /&gt;
&lt;br /&gt;
should print&lt;br /&gt;
&amp;lt;pre&amp;gt;nic:&lt;br /&gt;
        id=3000&lt;br /&gt;
        users=Cronk,aciss,adnan,ahoyleo,alexeizherdetsky,andrew4ta,aurele,bensonk,besler,brandond,cheelee,cholmes,chris,cmattson,ctompkins,ctownsend,cwise,cwoeck,dcronk,dongting,ehamovit,eric,erik-k,fchang,hammond,hoge,hoge_test,ivan,jacques,jhammond,jhou,jtg,kemerson,kmorris,kurtm,likai,lili,lorenz,lowd,mahshid,malony,mfatica,mmonil,msardell,naromero,ncascade,neuroapp,ntiller,nystrom,ozog,pgovyadi,raihan,rashawn,rmf,roessel,ryanm,sbrooks,scottb,sergei,smillst,speakless,swmgr,testcwoeck,vmware,weiler,wspear,wsvoorhees,yelle,znaika&lt;br /&gt;
        registry=LDAP&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This confirms that LDAP authentication is working. Horray! \o/&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chsec -f /etc/security/user -s default -a &amp;quot;SYSTEM=compat or LDAP&amp;quot;&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
because we have to have another step.&lt;br /&gt;
&lt;br /&gt;
=== Sudo setup ===&lt;br /&gt;
&lt;br /&gt;
visudo&lt;br /&gt;
&lt;br /&gt;
Enter&lt;br /&gt;
&amp;lt;pre&amp;gt;User_Alias      SUDO = erik-k,sameer,gansys,sivashan,nchaimov&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
for the user list and&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;SUDO ALL=(ALL) ALL&amp;lt;/pre&amp;gt; by root near the bottom&lt;br /&gt;
&lt;br /&gt;
=== NFS ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;chnfsdom .stor&lt;br /&gt;
startsrc -s nfsrgyd&lt;br /&gt;
nfso -p -o nfs_use_reserved_ports=1&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The file system mount creation commands:&lt;br /&gt;
&amp;lt;pre&amp;gt;mknfsmnt -f /home/users -d /vol/users -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&lt;br /&gt;
mknfsmnt -f /packages -d /mnt/packtree/aix72 -h 172.17.202.252 -M 'sys' -B -A -t rw -w bg -K 4 -k tcp&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point&lt;br /&gt;
&amp;lt;pre&amp;gt;startsrc -s nfs&amp;lt;/pre&amp;gt;&lt;br /&gt;
should work&lt;br /&gt;
&lt;br /&gt;
This will _massively_ simplify moving data back and forth!&lt;br /&gt;
&lt;br /&gt;
=== Spectrum Scale GPFS ===&lt;br /&gt;
&lt;br /&gt;
Copy Scale_DAE_install-5.1.2.0_pwraix.tar from ~erik-k/downloads to the machine.&lt;br /&gt;
&lt;br /&gt;
Create a directory and extract it (because it is a tarbomb!).&lt;br /&gt;
&lt;br /&gt;
ssh to root@ems1.stor, cat .ssh/id_rsa.pub, copy this to /root/.ssh/accepted_keys&lt;br /&gt;
&lt;br /&gt;
Check that root@ems1 can passwordless ssh to host.stor...&lt;br /&gt;
&lt;br /&gt;
mmaddnode -N HOSTNAME.stor:nonquorum::client --accept&lt;br /&gt;
&lt;br /&gt;
mmstartup -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
mmgetstate -N HOSTNAME&lt;br /&gt;
&lt;br /&gt;
.......&lt;br /&gt;
&lt;br /&gt;
[[Category:Procedures]]&lt;/div&gt;</summary>
		<author><name>Nic-wiki</name></author>
	</entry>
</feed>