Procedure:OpenshiftInstall - Revision history

Ekeever1 at 17:49, 21 October 2022

2022-10-21T17:49:01Z

← Older revision		Revision as of 17:49, 21 October 2022
Line 110:		Line 110:
	Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.		Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.

	~~root@vina~~:~# firewall-cmd --zone=public --direct --add-rule ipv4 nat POSTROUTING 0 -o enp1s0f0 -j MASQUERADE -d 128.223.202.0/24		'''UPDATE:'''

			The following firewalld commands will setup masquerading for the openshift subnet only,
			<pre>firewall-cmd --zone=public --direct --add-rule ipv4 nat POSTROUTING 0 -o enp1s0f0 -j MASQUERADE -d 128.223.202.0/24
			firewall-cmd --zone=public --direct --add-rule ipv4 nat POSTROUTING 0 -o enp1s0f0 -j MASQUERADE -s 172.17.100.0/24</pre>

			See: https://web.mit.edu/rhel-doc/4/RH-DOCS/rhel-sg-en-4/s1-firewall-ipt-fwd.html

	== Web server ==		== Web server ==

Ekeever1 at 17:38, 21 October 2022

2022-10-21T17:38:56Z

← Older revision		Revision as of 17:38, 21 October 2022
Line 109:		Line 109:

	Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.		Note that this configuration is insecure (it converts the host into a NAT proxy for ''the entire private ethernet'') and in reality the forwarding table should be restricted to the openshift cluster IP addresses.

			root@vina:~# firewall-cmd --zone=public --direct --add-rule ipv4 nat POSTROUTING 0 -o enp1s0f0 -j MASQUERADE -d 128.223.202.0/24

	== Web server ==		== Web server ==
Line 690:		Line 692:

	Now sit down and pour yourself a nice scotch, you deserve it.		Now sit down and pour yourself a nice scotch, you deserve it.

			= Operator configuration =

			== Authentication ==



			== Persistent storage ==



	[[Category:Procedures]]		[[Category:Procedures]]

Ekeever1 at 16:12, 18 October 2022

2022-10-18T16:12:17Z

← Older revision		Revision as of 16:12, 18 October 2022
Line 628:		Line 628:

	== Worker install on IBM s924 ==		== Worker install on IBM s924 ==

			If it has been more than a day since this was done, the ignition file's certificate needs to be refreshed,
			https://access.redhat.com/solutions/4799921

	First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).		First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).

	Then we must ssh to the VIOS and		Then we must ssh to the VIOS (user padmin) and
	<pre>oem_setup_env		<pre>oem_setup_env
	cd /Maingroup/images		cd /Maingroup/images

Ekeever1: /* Install master nodes */

2022-10-10T19:54:04Z

Install master nodes

← Older revision		Revision as of 19:54, 10 October 2022
Line 588:		Line 588:
	* Confirm haproxy is running and not dead (fails at boot due to selinux)		* Confirm haproxy is running and not dead (fails at boot due to selinux)
	* Confirm ethernet port is in trusted zone		* Confirm ethernet port is in trusted zone
			Note - this _should_ be fixed to not occur any more.

	== Install completion and bootstrap shutdown ==		== Install completion and bootstrap shutdown ==

Ekeever1: /* Install master nodes */

2022-10-10T19:53:22Z

Install master nodes

← Older revision		Revision as of 19:53, 10 October 2022
Line 579:		Line 579:
	* Balancer not running [TCP/SYN rejected]		* Balancer not running [TCP/SYN rejected]
	* Firewall misconfigured [TCP/SYN being dropped]		* Firewall misconfigured [TCP/SYN being dropped]

			Yet another problem is
			<pre>GET error: Get "https://api-int.openshift.stor:22623/config/master": dial tcp: lookup api-int.openshift.stor on [::1]:53: read udp [::1]:34055->[::1]:53: read: connection refused
			[* ] A start job is running for Ignition (fetch) (43s / no limit)[ 48.786901] ignition[847]: GET error: Get "https://api-int.openshift.stor:22623/config/master": dial tcp 172.17.100.100:22623: i/o timeout</pre>

			This occurs because the OC client node, which is also the load balancer for our configuration, has been restarted and returned to operation in an unhappy state. Namely, the firewall is not permitting the proxy to operate. Solution (check each in order):
			* Confirm selinux is off on client
			* Confirm haproxy is running and not dead (fails at boot due to selinux)
			* Confirm ethernet port is in trusted zone

	== Install completion and bootstrap shutdown ==		== Install completion and bootstrap shutdown ==

Ekeever1 at 19:41, 10 October 2022

2022-10-10T19:41:53Z

← Older revision		Revision as of 19:41, 10 October 2022
Line 503:		Line 503:
	Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.		Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.

	<pre>vina# virt-install \		For all I am able to determine (and this has happened _repeatedly_) virt-install will simply ignore size=X on the path...

			First create the virtual disk,
			<pre>cd /var/lib/libvirt/images
			qemu-img create -f qcow2 bootstrap.qcow2 120G</pre>

			This disk will not be preallocated (saving a good deal of space copying vast blocks of zeros around), and is simply assigned to the VM:

			<pre>
			OVER="4.11.2"
			vina# virt-install \
	--name bootstrap \		--name bootstrap \
	--vcpus 8 \		--vcpus 8 \
	--ram 16384 \		--ram 16384 \
	--disk path=/var/lib/libvirt/images/bootstrap.~~qcow2,size=20,format=~~qcow2,bus=virtio \		--disk path=/var/lib/libvirt/images/bootstrap.qcow2,bus=virtio \
	--graphics none \		--graphics none \
	--serial pty \		--serial pty \
	--console=pty,target_type=virtio \		--console=pty,target_type=virtio \
	--network network=ocp,model=virtio \		--network network=ocp,model=virtio \
	--extra-args "ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-~~4.6.8~~-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign " \		--extra-args "ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-${OVER}-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign " \
	--os-type linux --os-variant rhel7.0 \		--os-type linux --os-variant rhel7.0 \
	--location http://172.17.202.12:80/openshift_ppc64le/</pre>		--location http://172.17.202.12:80/openshift_ppc64le/</pre>
Line 526:		Line 536:
	Once the bootstrap node is online, we can initiate installation of the master nodes,		Once the bootstrap node is online, we can initiate installation of the master nodes,

	<pre>NODE=1		<pre>
			NODE=1
			OVER="4.11.2"

			cd /var/lib/libvirt/images
			qemu-img create -f qcow2 master${NODE}.qcow2 120G
	virt-install \		virt-install \
	--name master$NODE \		--name master$NODE \
Line 537:		Line 551:
	--console=pty,target_type=virtio \		--console=pty,target_type=virtio \
	--network network=ocp,model=virtio \		--network network=ocp,model=virtio \
	--extra-args "ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-~~4.6.8~~-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign " \		--extra-args "ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-${OVER}-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign " \
	--os-type linux --os-variant rhel7.0 \		--os-type linux --os-variant rhel7.0 \
	--location http://172.17.202.12/openshift_ppc64le/</pre>		--location http://172.17.202.12/openshift_ppc64le/</pre>

Nic-wiki: /* Worker install on IBM s924 */

2022-01-05T03:23:51Z

Worker install on IBM s924

← Older revision		Revision as of 03:23, 5 January 2022
Line 621:		Line 621:
	Then agonizingly 10-finger this into the vHMC terminal. This should run through a rapid Linux install, grab the rhcos image from Pliny (172.17.202.12), drop it onto the boot disk, then immediately restart.		Then agonizingly 10-finger this into the vHMC terminal. This should run through a rapid Linux install, grab the rhcos image from Pliny (172.17.202.12), drop it onto the boot disk, then immediately restart.

	Now interrupt the boot, shut the partition off, and remove the virtual optical disk so we do not pop back into the rhcos live boot.		Now interrupt the boot, shut the partition off, and remove the virtual optical disk so we do not pop back into the rhcos live boot. It will likely be necessary to interrupt the bootp and specify the attached vSCSI hard disk as the boot device.

	The partition SMS will now go through the bootp try/fail sequence, then drop into the bootloader that the loader on the virtual disk and initiate the "real" install.		The partition SMS will now go through the bootp try/fail sequence, then drop into the bootloader that the loader on the virtual disk and initiate the "real" install.

Nic-wiki: /* Worker install on IBM s924 */

2022-01-05T03:18:38Z

Worker install on IBM s924

← Older revision		Revision as of 03:18, 5 January 2022
Line 613:		Line 613:
	to copy the rhcos installer image to the vios, then use the system -> virtual storage -> vios -> manage -> virtual optical media to add it and assign to the rhcos LPAR.		to copy the rhcos installer image to the vios, then use the system -> virtual storage -> vios -> manage -> virtual optical media to add it and assign to the rhcos LPAR.

	Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS installer screen.		Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS live-installer grub screen.

	Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to terminal to substitute:		Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to a normal terminal to substitute:

	<pre>echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign</pre>		<pre>echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign</pre>

			Then agonizingly 10-finger this into the vHMC terminal. This should run through a rapid Linux install, grab the rhcos image from Pliny (172.17.202.12), drop it onto the boot disk, then immediately restart.

			Now interrupt the boot, shut the partition off, and remove the virtual optical disk so we do not pop back into the rhcos live boot.

			The partition SMS will now go through the bootp try/fail sequence, then drop into the bootloader that the loader on the virtual disk and initiate the "real" install.

	Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.		Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.

Nic-wiki: /* Load balancer */

2021-12-23T07:31:03Z

Load balancer

← Older revision		Revision as of 07:31, 23 December 2021
Line 311:		Line 311:
	=== Load balancer ===		=== Load balancer ===

	We install the HAProxy load balancer onto the client VM.		OpenShift requires some kind of load balancer to provide a central access point that mediates API access to the worker nodes. We install the HAProxy load balancer onto the client VM.

	In ~~a more performant situation~~, this ~~load~~ balancer would be ~~loaded onto a separate~~ (more powerful) VM, ~~onto~~ its own ~~entire~~ machine, or in ~~true~~ datacenter ~~applications would be~~ a ~~very expensive~~ piece of hardware.		In higher performance situations, this balancer would be its own (more powerful) VM, its own machine, or in a large-scale datacenter a piece of expensive hardware.

	Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,		Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,
Line 319:		Line 319:
	[ 93.245517] ignition[698]: GET error: Get "https://api-int.openshift.stor:22623/config/master": http: server gave HTTP response to HTTPS client</pre>		[ 93.245517] ignition[698]: GET error: Get "https://api-int.openshift.stor:22623/config/master": http: server gave HTTP response to HTTPS client</pre>

	The /etc/haproxy/haproxy.cfg file:		The /etc/haproxy/haproxy.cfg file we use:

	<pre>#---------------------------------------------------------------------		<pre>#---------------------------------------------------------------------

Nic-wiki: /* Worker install on IBM s924 */

2021-12-22T08:50:56Z

Worker install on IBM s924

← Older revision		Revision as of 08:50, 22 December 2021
Line 613:		Line 613:
	to copy the rhcos installer image to the vios, then use the system -> virtual storage -> vios -> manage -> virtual optical media to add it and assign to the rhcos LPAR.		to copy the rhcos installer image to the vios, then use the system -> virtual storage -> vios -> manage -> virtual optical media to add it and assign to the rhcos LPAR.

	Once this is done and we have verified that the lpar will boot		Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS installer screen.

	~~<pre>NODE=1~~		Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to terminal to substitute:
	~~virt-install \~~
	~~--name worker$NODE \~~		<pre>echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign</pre>
	~~--vcpus 32 \~~
	~~--ram 65536 \~~		Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.
	~~--disk path=/var/lib/libvirt/images/worker$~~NODE~~.qcow2~~,~~size=32,format=qcow2,bus=virtio \~~
	~~--graphics none \~~
	~~--serial pty \~~
	~~--console=pty,target_type=virtio \~~
	~~--network network=ocp,model=virtio \~~
	~~--extra-args "~~ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 ~~console=tty0 console=ttyS0~~ rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=~~vda~~ coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign ~~" \~~
	~~--os-type linux --os-variant rhel7.0 \~~
	~~--location http://172.17.202.12/openshift_ppc64le/~~</pre>

	== Approve new nodes ==		== Approve new nodes ==