Procedure:OnDemandSetup

This page lists various security permission that are involved with OOD,

https://osc.github.io/ood-documentation/latest/installation/modify-system-security.html

It is necessary to permit all to access ports 80, 443 and 5554 (the OOD authenticator)/tcp.

https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.

Step -1: Apparent working user access / bug bypass procedure [UPDATE: new version magically started working as expected]

Current status: It APPEARS that the OOD desktop launch process, in and of itself, works. The browser VNC viewer fails due to problems on ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,

==> httpd/access_ssl.log <==
128.223.202.205 - erik-k [08/Jul/2022:16:45:15 -0700] "GET /pun/sys/dashboard/batch_connect/sessions.js?_=1657318165288 HTTP/1.1" 200 5620 "https://ood.nic.uoregon.edu/pun/sys/dashboard/batch_connect/sessions" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
128.223.202.205 - - [08/Jul/2022:16:45:16 -0700] "GET /rnode/axis1/44232/websockify HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"

So clearly something is wrong with websockify setup on the server.

However: The normal login and desktop (not desktop-gnome) process succeeds and does launch a VNC viewer on axis1 (note - the cluster node setup process has only been carried out on axis1 until a procedure for a fully working environment is decrypted Thus this will only work if axis1 is idle, for the moment).

At this point, the user may SSH to orthus using

ssh orthus.nic.uoregon.edu -L3456:axis1.stor:590X

where X is determined from the connection.yml that is reachable via OOD by clicking your session ID once it starts. Then connect a VNC viewer to localhost:3456 with the password that is also in the connection.yml file.

Step 3 - Authentication setup

• set ssl cert file and keyfile at ood_config.yml:36
• under dex:, set ssl: true
• utilize most of their given template for LDAP connector, with some minor changes required for our ldap CN/DN/DCs
• Must set OIDCSSLValidateServer Off under httpd conf.d/ood-portal.conf after generating or it will not work (self-signed ldap1 cert)

At this point OOD is online/available and login works

Step 4 - cluster configuration

https://osc.github.io/ood-documentation/latest/installation/add-cluster-config.html

followed by slurm instructions at

https://osc.github.io/ood-documentation/latest/installation/resource-manager/slurm.html

per config documents, we need to setup munge/munged and the cluster munge key as well. It appears this system is able to access the cluster scheduler info outputs.

Step 5 - desktop configuration

https://osc.github.io/ood-documentation/master/enable-desktops/add-cluster.html

Cluster has been added to desktop environment settings for OOD, now complains about finding slurm. So Slurm must be installed on OOD!

Compilation & installation of slurm-21.08.4 on OOD at the same directory as on Orthus.

setting up slurm config on ood:

• must have munge-devel installed or it will silently fail to build munge support. One might perhaps speculate that this calls for something more than a tiny, buried not-even-an-error message given how critical munge is to 99% of slurm clusters...
• slurm install path must link to ../etc for slurm bins to find config info

Slurm now installed on OOD. Slurm configuration updated to have OOD in the slurm cluster.

Started mysql on orthus to get slurmdbd working because without it slurm does not work on ood...

Working at https://osc.github.io/ood-documentation/master/enable-desktops/add-cluster.html ,

Desktop will now attempt to start however we have an error in the *vnc launch command* of all things,

-> It turns out that OOD does not like tigervnc -> Install the turbovnc library and install turbovnc,

• Contents of /etc/yum.repos.d/TurboVNC.repo:

[TurboVNC]
name=TurboVNC official RPMs
baseurl=https://sourceforge.net/projects/turbovnc/files
gpgcheck=1
gpgkey=https://sourceforge.net/projects/turbovnc/files/VGL-GPG-KEY
       https://sourceforge.net/projects/turbovnc/files/VGL-GPG-KEY-1024
enabled=1

Still fails at

VNC_OUT=$(vncserver -log "vnc.log" -rfbauth "vnc.passwd" -nohttpd -noxstartup -geometry 800x600 -idletimeout 0  2>&1)

in the job script with

 Could not start Xvnc.

Unrecognized option: -nohttpd

Ugh, it appears this has been removed from turbovnc as of late. Grep Sledgehammer to the rescue,

[root@ood /]# grep nohttpd * -rn
opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.23/gems/ood_core-0.17.6/lib/ood_core/batch_connect/templates/vnc.rb:90:                VNC_OUT=$(vncserver -log "#{vnc_log}" -rfbauth "#{vnc_passwd}" -nohttpd -noxstartup #{vnc_args} 2>&1)
opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.23/gems/ood_core-0.18.1/lib/ood_core/batch_connect/templates/vnc.rb:90:                VNC_OUT=$(vncserver -log "#{vnc_log}" -rfbauth "#{vnc_passwd}" -nohttpd -noxstartup #{vnc_args} 2>&1)

Removing these and will re-try once axis1 is unused (I have only test installed the turbovnc system on axis1).

Okay, tested with -nohttpd removed and some progress,

Install python3-websockify package.

websockify

https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/modify-cluster-configuration.html

This does actually need to be modified. Current content of /etc/ood/config/clusters.d/axis.yml,

v2:
  metadata:
    title: "Axis cluster"
  login:
    host: "orthus.nic.uoregon.edu"
  job:
    adapter: "slurm"
    cluster: "axis"
    bin: "/opt/slurm/current/bin"
    conf: "/opt/slurm/etc/slurm.conf"
  batch_connect:
    vnc:
      script_wrapper: |
        module purge
        export PATH="/opt/TurboVNC/bin:$PATH"
        export WEBSOCKIFY_CMD="/usr/local/bin/websockify"
        %s

This sets the path to turbovnc and websockify.

Update: The correct (yum) installation of python3-websockify puts it in /usr/bin. Because existing config specs /usr/local/bin, must symlink

gnome desktop

Under /etc/ood/config/apps/bc_desktop/ must create a new axis_gnome.yml that sets the default desktop to gnome not mate,

title: "Axis | Gnome Desktop"
cluster: "axis"
form:
  - desktop
attributes:
  desktop: "gnome"

This gets us far enough that the desktop does actually claim to start and run which is further than we've ever gotten before.