Procedure:OnDemandSetup

This page lists various security permission that are involved with OOD,

https://osc.github.io/ood-documentation/latest/installation/modify-system-security.html

It is necessary to permit all to access ports 80, 443 and 5554 (the OOD authenticator)/tcp.

https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.

Step -1: Apparent working user access / bug bypass procedure [UPDATE: new version magically started working as expected]

Current status: It APPEARS that the OOD desktop launch process, in and of itself, works. The browser VNC viewer fails due to problems on ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,

==> httpd/access_ssl.log <==
128.223.202.205 - erik-k [08/Jul/2022:16:45:15 -0700] "GET /pun/sys/dashboard/batch_connect/sessions.js?_=1657318165288 HTTP/1.1" 200 5620 "https://ood.nic.uoregon.edu/pun/sys/dashboard/batch_connect/sessions" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
128.223.202.205 - - [08/Jul/2022:16:45:16 -0700] "GET /rnode/axis1/44232/websockify HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"

So clearly something is wrong with websockify setup on the server.

However: The normal login and desktop (not desktop-gnome) process succeeds and does launch a VNC viewer on axis1 (note - the cluster node setup process has only been carried out on axis1 until a procedure for a fully working environment is decrypted Thus this will only work if axis1 is idle, for the moment).

At this point, the user may SSH to orthus using

ssh orthus.nic.uoregon.edu -L3456:axis1.stor:590X

where X is determined from the connection.yml that is reachable via OOD by clicking your session ID once it starts. Then connect a VNC viewer to localhost:3456 with the password that is also in the connection.yml file.

Step 3 - Authentication setup

• set ssl cert file and keyfile at ood_config.yml:36
• under dex:, set ssl: true
• utilize most of their given template for LDAP connector, with some minor changes required for our ldap CN/DN/DCs
• Must set OIDCSSLValidateServer Off under httpd conf.d/ood-portal.conf after generating or it will not work (self-signed ldap1 cert)

At this point OOD is online/available and login works

Step 4 - cluster configuration

https://osc.github.io/ood-documentation/latest/installation/add-cluster-config.html

followed by slurm instructions at

https://osc.github.io/ood-documentation/latest/installation/resource-manager/slurm.html

per config documents, we need to setup munge/munged and the cluster munge key as well. It appears this system is able to access the cluster scheduler info outputs.

setting up slurm config on ood:

• must have munge-devel installed or it will silently fail to build munge support. One might perhaps speculate that this calls for something more than a tiny, buried not-even-an-error message given how critical munge is to 99% of slurm clusters...
• slurm install path must link to ../etc for slurm bins to find config info

Slurm is built at /opt/slurm/version__ as on other systems.

Step 5 - cluster node software setup

Two nonstandard packages are needed on the remote desktop hosts, tigervnc and websockify.

Repo for turbovnc:

[root@voltar yum.repos.d]# cat TurboVNC.repo 
[TurboVNC]
name=TurboVNC official RPMs
baseurl=https://sourceforge.net/projects/turbovnc/files
gpgcheck=1
gpgkey=https://sourceforge.net/projects/turbovnc/files/VGL-GPG-KEY
       https://sourceforge.net/projects/turbovnc/files/VGL-GPG-KEY-1024
enabled=1

Commands:

yum install python3-websockify turbovnc
# The below is due to how ood cluster launcher script on OOD at root@ood3:/etc/ood/config/clusters.d is configured
# which is in turn because of how websockify ended up installed on the axis nodes so we're default stuck with it now
cd /usr/local/bin
ln -s /usr/bin/websockify ./websockify

These will install the needed packages to make the thing go.

Step 5 - desktop configuration on the cluster

https://osc.github.io/ood-documentation/master/enable-desktops/add-cluster.html

Desktop will now attempt to start however we have an error in the *vnc launch command* of all things,

-> It turns out that OOD does not like tigervnc -> Install TurboVNC

By default it will fail at

VNC_OUT=$(vncserver -log "vnc.log" -rfbauth "vnc.passwd" -nohttpd -noxstartup -geometry 800x600 -idletimeout 0  2>&1)

in the job script with

 Could not start Xvnc.

Unrecognized option: -nohttpd

Ugh, it appears this has been removed from turbovnc as of late. Grep Sledgehammer to the rescue,

[root@ood /]# grep nohttpd * -rn
opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.23/gems/ood_core-0.17.6/lib/ood_core/batch_connect/templates/vnc.rb:90:                VNC_OUT=$(vncserver -log "#{vnc_log}" -rfbauth "#{vnc_passwd}" -nohttpd -noxstartup #{vnc_args} 2>&1)
opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.23/gems/ood_core-0.18.1/lib/ood_core/batch_connect/templates/vnc.rb:90:                VNC_OUT=$(vncserver -log "#{vnc_log}" -rfbauth "#{vnc_passwd}" -nohttpd -noxstartup #{vnc_args} 2>&1)

The invocation now works.

Next step at https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/modify-cluster-configuration.html does need to be modified.

This does actually need to be modified. Current content of /etc/ood/config/clusters.d/axis.yml,

v2:
  metadata:
    title: "Axis cluster"
  login:
    host: "orthus.nic.uoregon.edu"
  job:
    adapter: "slurm"
    cluster: "axis"
    bin: "/opt/slurm/current/bin"
    conf: "/opt/slurm/etc/slurm.conf"
  batch_connect:
    basic:
      script_wrapper: |
        module purge
        %s
      set_host: "host=$(hostname -A | awk '{print $1}')"
    vnc:
      script_wrapper: |
        module purge
        export PATH="/opt/TurboVNC/bin:$PATH"
        export WEBSOCKIFY_CMD="/usr/local/bin/websockify"
        %s
      set_host: "host=$(hostname -A | awk '{print $1}')"

This sets the path to turbovnc and websockify to make sure they launch and sets the hostname (out of those that can be chosen).

The correct (yum) installation of python3-websockify puts it in /usr/bin/websockify. Because existing config specs /usr/local/bin, just symlink

gnome desktop

Under /etc/ood/config/apps/bc_desktop/ must create a new axis_gnome.yml that sets the default desktop to gnome not mate,

title: "Axis | Gnome Desktop"
cluster: "axis"
form:
  - desktop
attributes:
  desktop: "gnome"