Procedure:OpenshiftInstall

This page documents the long, painful and complex process of installing OpenShift in a manner that will hopefully reveal the numerous rakes in the grass and avoid a repeat of the famous Sideshow Bob scene.

The big picture:

asdfasdfasdf

Outside guidance URLs

• The ostensible guide: https://docs.openshift.com/container-platform/4.6/installing/installing_bare_metal/installing-bare-metal.html#installation-obtaining-installer_installing-bare-metal
  • This reads a lot like a semi-organized stream of consciousness, simply proceeding from A to B to C with no indexing and few explanations.
• https://cloud.redhat.com/openshift/downloads
  • This leads to the openshift mirror repos where the openshift_client and openshift_installer tar files are found
• https://mirror.openshift.com/pub/openshift-v4/ppc64le/dependencies/rhcos
  • PLEASE NOTE: It is absolutely CRITICAL to have the SAME versions of the client, installer and rhcos
  • Navigate specifically to version/xx, do not just grab 'stable'.
• https://www.redhat.com/en/blog/installing-openshift-41-using-libvirt-and-kvm
  • This blog page provides some useful guidance and especially a useful template for the HAProxy configuration

I note that this document describes the installation of a specific and isolated OC cluster.

The OpenShift client system is itself a small VM named client.openshift.stor; This client also runs the proxy/load balancer.

External prerequisites

Openshift requires several services/components external to itself in order to work. These are,

• Web server: for serving up open RHCOS images and Ignition files to the installer
• Ethernet virbr: As this instance considers the installation of the entire cluster onto VMs living on a single real host, a virtual ethernet bridge must be established to facilitate communication
• Ethernet NAT masquerade: The cluster lives on a private LAN, and in this case the host machine is configured to provide NAT service to the OC virtual machines
• DNS: Openshift requires for itself a subdomain and a certain set of forward- and reverse-defined DNS entries.
• Load balancer: In this case, we install HAProxy on the client system and utilize it as the balancer. Alternatively, a dedicated balancer (VM, or machine, or hardware appliance) may be used.

Topology Foreword

For clarity: The topology of connectivity utilized is

(insert image here)

Ethernet connectivity

The ethernet setup required takes two steps. First, a virtual ethernet bridge (software Level 2) must be setup on each VM host's private ethernet interface. This will create a br0 interface for the virtual bridge. The hardware en___ interface will lose its IP and be slaved to the bridge, and the br0 interface will acquire the IP.

Now, qemu-kvm virtual machines will be able to connect to the br0 bridge, which (from their perspective) is as good as being on the physical Ethernet switch.

As the second step, an IP masquerade/NAT is established for the cluster IPs such that they are able to name the host IP as their gateway and reach the wider Internet (to download updates and containers).

Bridge setup

Creating the virbr is simple on an Ubuntu host,

root@vina:~# cat /etc/netplan/00-installer-config.yaml 
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp1s0f0:
      dhcp4: true
    enp1s0f1:
#      dhcp4: true
      mtu: 9000
  version: 2
  bridges:
    br0:
      interfaces: [enp1s0f1]
      dhcp4: true
      mtu: 9000

Note that here, mtu=9000 is set because our storage/private ethernet is setup to use jumbo frames and it is quite necessary that the IP MTUs match, or magic packet loss is likely to occur; Software bridges may automatically fragment jumbo frames, but hardware will simply discard overlength packets.

The bridging setup on RHEL can be achieved several ways. I prefer to just punch it straight into /etc/sysconfig/network-scripts/*, as in this example that sets up one of the virbrs on our infrastructure vm node,

[root@orion network-scripts]# cat ifcfg-eno1
TYPE=Ethernet
NAME=eno1-bridge-slave
UUID=9e7e3a89-3358-4830-a033-0a4154c68c55
DEVICE=eno1
ONBOOT=yes
BRIDGE=br0
HWADDR=3c:ec:ef:1a:71:3e

[root@orion network-scripts]# cat ifcfg-br0
STP=no
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
UUID=3549a392-12ac-4c7f-bdd5-4e86d8654ddf
DEVICE=br0
ONBOOT=yes
MTU=9000

It may be possible to leave STP enabled, however this was found to cause problems in some settings (the interface claims to be activating forever and never enters forwarding - i.e. the host physical ethernet port did not talk STP back)

Once the bridge has been defined, we must inform kvm about it by defining an xml, importing it and marking it to autostart in kvm.

IP masquerading

Assuming that routing is enabled on the host and normal routing table entries are setup, the following three lines will setup IP masquerade between the public interface ('ifpublic') and br0,

iptables -A FORWARD -i br0 -o ifpublic -j ACCEPT
iptables -A FORWARD -i ifpublic -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -o enp1s0f0 -j MASQUERADE

Note that this configuration is insecure (it converts the host into a NAT proxy for the entire private ethernet) and in reality the forwarding table should be restricted to the openshift cluster IP addresses.

UPDATE:

The following firewalld commands will setup masquerading for the openshift subnet only,

firewall-cmd --zone=public --direct --add-rule ipv4 nat POSTROUTING 0 -o enp1s0f0  -j MASQUERADE -d 128.223.202.0/24 
firewall-cmd --zone=public --direct --add-rule ipv4 nat POSTROUTING 0 -o enp1s0f0  -j MASQUERADE -s 172.17.100.0/24

See: https://web.mit.edu/rhel-doc/4/RH-DOCS/rhel-sg-en-4/s1-firewall-ipt-fwd.html

Web server

Some web space must be provisioned that can serve up

• RHCOS images
• Ignition files

The system at this stage is not paranoid; In our case, I have setup a 10-openshift.conf Apache server bound to Pliny's private network interface (172.17.202.12). Stripping all commentary from the configuration file,

<VirtualHost 172.17.202.12:80>
    ServerAdmin systems@nic.uoregon.edu
    ServerName pliny.nic.uoregon.edu

    DocumentRoot /home/web_openshift/

    # if not specified, the global error log is used
    ErrorLog /var/log/httpd/openshift/error_log
    CustomLog /var/log/httpd/openshift/access_log combined

    HostnameLookups Off
    UseCanonicalName Off
    ServerSignature On
    DirectoryIndex index.html

    <Location />
        options +indexes
        <RequireAny>
            Require ip 172.17.0.0/16
        </RequireAny>
    </Location>
</VirtualHost>

<VirtualHost 172.17.202.12:443>
    #  General setup for the virtual host
    DocumentRoot "/home/web_openshift/"
    ServerName pliny.nic.uoregon.edu

    ErrorLog /var/log/httpd/openshift/ssl_error_log
    TransferLog /var/log/httpd/openshift/ssl_access_log

    SSLEngine on

    #   SSL Protocol Support:
    SSLProtocol All -SSLv2 -SSLv3
    SSLCipherSuite    ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    SSLHonorCipherOrder     on

    SSLCertificateFile **********
    SSLCertificateKeyFile **********
    SSLCertificateChainFile **********

    <Files ~ "\.(cgi|shtml|phtml|php3?)$">
        SSLOptions +StdEnvVars
    </Files>
    <Directory "/var/www/www/cgi-bin">
        SSLOptions +StdEnvVars
    </Directory>

    SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

    CustomLog /var/log/httpd/ssl_request_log   ssl_combined

    HostnameLookups Off
    UseCanonicalName Off
    ServerSignature On
    DirectoryIndex index.html index.htm index.php
    Options +indexes

    <Location />
        <RequireAny>
            Require ip 172.17.0.0/16
        </RequireAny>
    </Location>
</VirtualHost>

I have decided to at least organize by processor architecture (a cluster must be 100% x86_64 or ppc64le, no mixing). The basic data that must be present in the web directory is as follows,

[root@pliny web_openshift]# ls /home/web_openshift/
openshift_ppc64le
[root@pliny web_openshift]# ls -la /home/web_openshift/openshift_ppc64le/
total 1787720
drwxr-xr-x. 3 root root      4096 Jul 27 19:48 .
drwxr-xr-x. 3 root root        31 Jul 27 19:07 ..
-rw-r--r--. 1 root root    288355 Jul 27 19:06 bootstrap.ign
-rw-r--r--. 1 root root      1716 Jul 27 19:06 master.ign
drwxr-xr-x. 2 root root        63 Jul 27 19:06 old
-rw-r--r--. 1 root root  80882648 Dec 15  2020 rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img
-rw-r--r--. 1 root root  26903229 Dec 15  2020 rhcos-4.6.8-ppc64le-live-kernel-ppc64le
-rw-r--r--. 1 root root 918583296 Dec 15  2020 rhcos-4.6.8-ppc64le-live.ppc64le.iso
-rw-r--r--. 1 root root 803940864 Dec 15  2020 rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img
-rw-r--r--. 1 root root       354 Jul 27 19:08 .treeinfo
-rw-r--r--. 1 root root      1716 Jul 27 19:06 worker.ign

Note that the .ign files generated by openshift_install are, by default, chmod 600. If these are copied to the web server without resetting to 644 (go+r), it will fail to serve them up and the installer will explode on the launchpad without generating useful error output. My prepare_install.sh script performs this change automatically if used.

Certain information must be present in the .treeinfo file:

[root@pliny openshift_ppc64le]# cat .treeinfo 
[general]
name = CentOS-7
family = CentOS
timestamp = 1587405659.3
variant =
version = 7
packagedir =
arch = ppc64le

[stage2]
mainimage = rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img

[images-ppc64le]
kernel = rhcos-4.6.8-ppc64le-live-kernel-ppc64le
initrd = rhcos-4.6.8-ppc64le-live-initramfs.ppc64le.img
boot.iso = rhcos-4.6.8-ppc64le-live.ppc64le.iso

Of course, the version given (4.6.8 here) needs to match the files actually present.

DNS setup

Our DNS server identifies several private TLDs, including one (.stor) for the private ethernet interfaces of our nodes, for OACISS-local IP addresses. As our openshift setup will not be publicly reachable, we make it live entirely on this private TLD.

First, we create for it the openshift.stor domain within the DNS private view,

zone "openshift.stor" in {
        type master;
        masterfile-format text;
        file "openshift_forward.zone";
        allow-update { none; };
        allow-transfer { private_servers; };
        allow-query { trusted_nets; };
};

As we can only define one reverse lookup table for the 172.17.0.0/16 network, the reverse records must go under the storage.reverse file.

The openshift_forward.zone file:

$TTL 2h
@                       IN      SOA     ns.nic.local. systems.nic.uoregon.edu. (
                                        2021072706      ; Serial number
                                        21600           ; Refresh(6hrs)
                                        1800            ; Retry(30min)
                                        1209600         ; Expire(2wks)
                                        432000 )        ; Minimum(5dys)
; vim: ts=4:
; Name servers.

                        IN      NS      fripp.nic.local.

; Openshift virtual machines
client                  IN      A       172.17.100.100
api                     IN      A       172.17.100.100
api-int                 IN      A       172.17.100.100
*.apps                  IN      A       172.17.100.100

bootstrap               IN      A       172.17.100.110

master1                 IN      A       172.17.100.101
master2                 IN      A       172.17.100.102
master3                 IN      A       172.17.100.103
worker1                 IN      A       172.17.100.104
worker2                 IN      A       172.17.100.105
worker3                 IN      A       172.17.100.106
worker4                 IN      A       172.17.100.107
worker5                 IN      A       172.17.100.108
worker6                 IN      A       172.17.100.109

And the relevant entries in storage_reverse.zone:

; OpenSHIFT virtual machine reverse lookups
101.100         IN      PTR     master1.openshift.stor.
102.100         IN      PTR     master2.openshift.stor.
103.100         IN      PTR     master3.openshift.stor.

104.100         IN      PTR     worker1.openshift.stor.
105.100         IN      PTR     worker2.openshift.stor.
106.100         IN      PTR     worker3.openshift.stor.
107.100         IN      PTR     worker4.openshift.stor.
108.100         IN      PTR     worker5.openshift.stor.
109.100         IN      PTR     worker6.openshift.stor.

Note, as is easily forgotten, that the reverse lookups must terminate with .stor. and not just .stor or reverse resolution does not work the way you expect it to :)

The above establishes hostnames for three masters in a quorum and provisions hostnames for up to six workers.

Do not forget to increment the dns serial numbers when this is edited!

Client VM

As the initial entry in the OpenShift saga, we create a virtual machine named (cunningly) client from a pulled down Centos 8.3 live dvd image,

virt-install --virt-type=kvm --name client --memory 2048 --vcpus=2 --os-variant=rhel8.3 --cdrom=/var/lib/libvirt/boot/CentOS-8.3.2011-ppc64le-dvd1.iso --network=network=ocp,model=virtio --console=pty,target_type=virtio --disk path=/var/lib/libvirt/images/centos8.qcow2,size=20,bus=virtio,format=qcow2 --serial pty --graphics none

This will be the access point for the cluster as well as (in our case) running the load balancer.

Before going any further, best to setup the ssh key as this will be needed shortly:

ssh-keygen  -t ed25519

Load balancer

OpenShift requires some kind of load balancer to provide a central access point that mediates API access to the worker nodes. We install the HAProxy load balancer onto the client VM.

In higher performance situations, this balancer would be its own (more powerful) VM, its own machine, or in a large-scale datacenter a piece of expensive hardware.

Our situation finds the openshift cluster living on a single ethernet LAN, such that the job of haproxy is just to round-robin requests. It is critical that the 'mode http' be removed from the 'global' section of the default config file. If it is not, the VM install process will stall forever with this error,

[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22
[   93.245517] ignition[698]: GET error: Get "https://api-int.openshift.stor:22623/config/master": http: server gave HTTP response to HTTPS client

The /etc/haproxy/haproxy.cfg file we use:

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   https://www.haproxy.org/download/1.8/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          30s
    timeout server          30s
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend kubernetes_api
    bind 172.17.100.100:6443
    default_backend kubernetes_api

backend kubernetes_api
    balance roundrobin
    option ssl-hello-chk
    server bootstrap bootstrap.openshift.stor:6443 check
    server master1 master1.openshift.stor:6443 check
    server master2 master2.openshift.stor:6443 check
    server master3 master3.openshift.stor:6443 check

frontend machine_config
    bind 172.17.100.100:22623
    default_backend machine_config

backend machine_config
    balance roundrobin
    option ssl-hello-chk
    server bootstrap bootstrap.openshift.stor:22623 check
    server master1 master1.openshift.stor:22623 check
    server master2 master2.openshift.stor:22623 check
    server master3 master3.openshift.stor:22623 check

frontend router_https
    bind 172.17.100.100:443
    default_backend router_https

backend router_https
    balance roundrobin
    option ssl-hello-chk
    server worker1 worker1.openshift.stor:443 check
    server worker2 worker2.openshift.stor:443 check

frontend router_http
    mode http
    option httplog
    bind 172.17.100.100:80
    default_backend router_http

backend router_http
    mode http
    balance roundrobin
    server worker1 worker1.openshift.stor:80 check
    server worker2 worker2.openshift.stor:80 check

SElinux and firewall compatibility instructions are mandatory at this juncture:

semanage  port -a -t http_port_t -p tcp 22623
semanage  port -a -t http_port_t -p tcp 6443
firewall-cmd --add-port=6443/tcp
firewall-cmd --add-port=22623/tcp
firewall-cmd --runtime-to-permanent

Note, this assumes (correctly in the OACISS case) that the client.openshift.stor VM has only a single interface which it default places in the 'public' zone.

Setup process

Ignition files

After downloading

openshift-client-linux-4.6.8.tar.gz

and

openshift-install-linux-4.6.8.tar.gz

and unpacking them in /root/OCP/ on the client, it is time to generate the Ignition files that will automagically configure the virtual machines.

These are created by openshift-install after reading a .yaml configuration file.

This is the yaml given on the openshift install site as an example of a minimal configuration. Because we are installing the workers manually we must state 0 replicas for workers. Note that openshift-install helpfully deletes the input configuration yaml file, so this backup must be copied each time it is run...

[root@client OCP]# cat install-config.yaml.bak 
apiVersion: v1
baseDomain: stor
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: openshift
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
fips: false
pullSecret: '{}'
sshKey: 'ssh-ed25519 AAAA************ root@client.openshift.stor'

Naturally, of course, the actual pullSecret and client SSH root trust key should be present.

I have packaged the sequence of steps required next into a helpful prepare_install.sh script,

[root@client OCP]# cat prepare_install.sh p
#!/bin/bash

echo "Deleting existing install logs and ign files"
rm -rf .openshift_install.log .openshift_install_state.json metadata.json bootstrap.ign worker.ign master.ign auth

echo "cp install-config.yaml.bak install-config.yaml"
cp install-config.yaml.bak install-config.yaml

echo "creating manifests"
./openshift-install create manifests --dir=./

echo "creating Ignition config files"
./openshift-install create ignition-configs --dir=./

echo "Copying to web server on Pliny"
chmod go+r *ign
scp *ign root@pliny:/home/web_openshift/

cp -f /root/OCP/auth/kubeconfig /root/.kube/config

Install bootstrap node

Once the ignition files are ready on the web server (consider checking the directory with lynx!), we can stand up the bootstrap machine. This will take over the console for a while so it is best to do it in a separate terminal.

For all I am able to determine (and this has happened _repeatedly_) virt-install will simply ignore size=X on the path...

First create the virtual disk,

cd /var/lib/libvirt/images
qemu-img create -f qcow2 bootstrap.qcow2 120G

This disk will not be preallocated (saving a good deal of space copying vast blocks of zeros around), and is simply assigned to the VM:

OVER="4.11.2"
vina# virt-install \
--name bootstrap \
--vcpus 8 \
--ram 16384 \
--disk path=/var/lib/libvirt/images/bootstrap.qcow2,bus=virtio \
--graphics none \
--serial pty \
--console=pty,target_type=virtio \
--network network=ocp,model=virtio \
--extra-args "ip=172.17.100.110::172.17.202.79:255.255.0.0:bootstrap.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12:80/openshift_ppc64le/rhcos-${OVER}-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12:80/openshift_ppc64le/bootstrap.ign " \
--os-type linux --os-variant rhel7.0 \
--location http://172.17.202.12:80/openshift_ppc64le/

It will take something like 4 minutes for this to run and crank the bootstrap machine.

Once this is done, ssh from client to core@bootstrap.openshift.stor and run 'journalctl | grep -i expired', and hopefully no output appears.

If this succeeds, proceed.

Install master nodes

Once the bootstrap node is online, we can initiate installation of the master nodes,

NODE=1
OVER="4.11.2"

cd /var/lib/libvirt/images
qemu-img create -f qcow2 master${NODE}.qcow2 120G
virt-install \
--name master$NODE \
--vcpus 8 \
--ram 16384 \
--disk path=/var/lib/libvirt/images/master$NODE.qcow2,size=32,format=qcow2,bus=virtio \
--graphics none \
--serial pty \
--console=pty,target_type=virtio \
--network network=ocp,model=virtio \
--extra-args "ip=172.17.100.10$NODE::172.17.202.79:255.255.0.0:master$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-${OVER}-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/master.ign " \
--os-type linux --os-variant rhel7.0 \
--location http://172.17.202.12/openshift_ppc64le/

The $NODE substitution increments the virtual disk name, IP and hostname appropriately. We, here, have just the three master nodes as 100.101, 2 and 3.

These should take 5-ish minutes to install, boot, self-update and reboot before they try and contact the hive mind.

Several problems can manifest at this point, all caused by misconfiguration on the load balancer,

[   88.232297] ignition[698]: GET error: Get "https://api-int.openshift.stor:22623/config/master": http: server gave HTTP response to HTTPS client
[   ***] A start job is running for Ignition (fetch) (1min 30s / no limit)[   93.232888] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #22
[   93.245517] ignition[698]: GET error: Get "https://api-int.openshift.stor:22623/config/master": http: server gave HTTP response to HTTPS client

The problem lies in the haproxy configuration file. Under 'global' do not have 'mode http'. This is given correctly in the haproxy config above.

[   ***] A start job is running for Ignition (fetch) (31min 30s / no limit)[ 1893.903489] ignition[698]: GET https://api-int.openshift.stor:22623/config/master: attempt #381
[ 1893.921039] ignition[698]: GET error: Get "https://api-int.openshift.stor:22623/config/master": x509: certificate signed by unknown authority

This occurred when, in initially configuring haproxy, I accidentally told the forwarder for port 22623 to talk to master*:6443.

Another possible problem is

[***   ] A start job is running for Ignition (fetch) (1min 45s / no limit)[  108.236817] ignition[690]: GET https://api-int.openshift.stor:22623/config/master: attempt #25
[  108.249306] ignition[690]: GET error: Get "https://api-int.openshift.stor:22623/config/master": dial tcp 172.17.100.100:22623: connect: no route to host

This error is provoked for several possible reasons relating to the load balancer setup (See load balancer section). Most likely either

• Balancer not running [TCP/SYN rejected]
• Firewall misconfigured [TCP/SYN being dropped]

Yet another problem is

GET error: Get "https://api-int.openshift.stor:22623/config/master": dial tcp: lookup api-int.openshift.stor on [::1]:53: read udp [::1]:34055->[::1]:53: read: connection refused
[*     ] A start job is running for Ignition (fetch) (43s / no limit)[   48.786901] ignition[847]: GET error: Get "https://api-int.openshift.stor:22623/config/master": dial tcp 172.17.100.100:22623: i/o timeout

This occurs because the OC client node, which is also the load balancer for our configuration, has been restarted and returned to operation in an unhappy state. Namely, the firewall is not permitting the proxy to operate. Solution (check each in order):

• Confirm selinux is off on client
• Confirm haproxy is running and not dead (fails at boot due to selinux)
• Confirm ethernet port is in trusted zone

Note - this _should_ be fixed to not occur any more.

Install completion and bootstrap shutdown

From the OCP directory on the client, once the master nodes launch into their self-setup process, run

./openshift-install --dir=./ wait-for bootstrap-complete --log-level=info

This command will block your terminal window until the master nodes finish setting themselves up (a 10+ minute endeavour in my case), thereupon informing you it is safe to shut the bootstrap machine down;

virsh shutdown bootstrap

At this point, running

[root@client OCP]# ./oc get co

should vomit out 30 or so lines of the form "NAME [same version] True ...".

If we get nodes, we now (technically) have a working cluster,

[root@client OCP]# ./oc get nodes 
NAME                     STATUS   ROLES           AGE    VERSION
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803

Worker install

Now it is time to install the worker nodes. This may be done on the same machine, or across whatever real machines are going to run the cluster,

NODE=1
virt-install \
--name worker$NODE \
--vcpus 32 \
--ram 65536 \
--disk path=/var/lib/libvirt/images/worker$NODE.qcow2,size=32,format=qcow2,bus=virtio \
--graphics none \
--serial pty \
--console=pty,target_type=virtio \
--network network=ocp,model=virtio \
--extra-args "ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 console=tty0 console=ttyS0 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign " \
--os-type linux --os-variant rhel7.0 \
--location http://172.17.202.12/openshift_ppc64le/

Like the install commands for master nodes, the commands for the workers are clones, save for incrementing the IP addresses, workerN hostnames and the virtual disk names.

Worker install on IBM s924

If it has been more than a day since this was done, the ignition file's certificate needs to be refreshed, https://access.redhat.com/solutions/4799921

First an LPAR must be created and allocated processors, memory and a virtual disk from the available pool (See vHMC setup procedure).

Then we must ssh to the VIOS (user padmin) and

oem_setup_env
cd /Maingroup/images
scp erik-k@172.17.202.18:~/downloads/rhcos-4.6.8-ppc64le-live.ppc64le.iso ./

to copy the rhcos installer image to the vios, then use the system -> virtual storage -> vios -> manage -> virtual optical media to add it and assign to the rhcos LPAR.

Once this is done and we have verified that the lpar will boot, it can be started and will immediately pop into the RHCOS live-installer grub screen.

Interrupt it and enter a custom kernel command line. setting NODE first, paste the following to a normal terminal to substitute:

echo ip=172.17.100.10$(expr 3 + $NODE)::172.17.202.79:255.255.0.0:worker$NODE.openshift.stor::none nameserver=172.17.202.25 rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.live.rootfs_url=http://172.17.202.12/openshift_ppc64le/rhcos-4.6.8-ppc64le-live-rootfs.ppc64le.img coreos.inst.ignition_url=http://172.17.202.12/openshift_ppc64le/worker.ign

Then agonizingly 10-finger this into the vHMC terminal. This should run through a rapid Linux install, grab the rhcos image from Pliny (172.17.202.12), drop it onto the boot disk, then immediately restart.

Now interrupt the boot, shut the partition off, and remove the virtual optical disk so we do not pop back into the rhcos live boot. It will likely be necessary to interrupt the bootp and specify the attached vSCSI hard disk as the boot device.

The partition SMS will now go through the bootp try/fail sequence, then drop into the bootloader that the loader on the virtual disk and initiate the "real" install.

Eventually new CSRs will manifest and need to be acknowledged (see below) to add the node to the cluster.

Approve new nodes

Once the consoles for the worker nodes are sitting at the login prompt, we can add them to the cluster.

Running

./oc get csr

will show that we have two key requests waiting from the workers. Do

./oc adm certificate approve $NAME

for each of the two NAMEd requests to inject the nanoprobes and make them part of the collective.

After about 10-15 seconds we can get nodes again and see they have appeared,

[root@client OCP]# ./oc get nodes 
NAME                     STATUS   ROLES           AGE    VERSION
master1.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803
master2.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803
master3.openshift.stor   Ready    master,worker   35m    v1.19.0+7070803
worker1.openshift.stor   Ready    worker          109s   v1.19.0+7070803
worker2.openshift.stor   Ready    worker          102s   v1.19.0+7070803

We note that the master nodes are schedulable because we originally set ourselves up with no worker nodes. Running

./oc edit schedulers.config.openshift.io cluster

And edit the line near the bottom for master schedulable from 'true' to 'false'. Now we have what we want:

[root@client OCP]# ./oc get nodes 
NAME                     STATUS   ROLES    AGE   VERSION
master1.openshift.stor   Ready    master   44m   v1.19.0+7070803
master2.openshift.stor   Ready    master   44m   v1.19.0+7070803
master3.openshift.stor   Ready    master   44m   v1.19.0+7070803
worker1.openshift.stor   Ready    worker   10m   v1.19.0+7070803
worker2.openshift.stor   Ready    worker   10m   v1.19.0+7070803

Post-install smart moves

Now that it's working, another very minor not at all noteworthy thing to mention...

If the system ever goes down for more than 24 hr, it will be impossible to restart. So, now that it's working, this would be a great time to go run 'virsh list' and then 'virsh autostart X' all the domains: client, master[1 | 2 | 3] and worker[1 | 2] to make sure they come back up even if the host restarts.

Now sit down and pour yourself a nice scotch, you deserve it.