Procedure:OnDemandSetup - Revision history

Ekeever1: /* Step 5 - desktop configuration on the cluster */

2025-03-21T17:21:41Z

Step 5 - desktop configuration on the cluster

← Older revision		Revision as of 17:21, 21 March 2025
Line 63:		Line 63:
	These will install the needed packages to make the thing go.		These will install the needed packages to make the thing go.

	== Step 5 - desktop configuration on ~~the cluster~~ ==		== Step 5 - desktop configuration on OOD ==

	https://osc.github.io/ood-documentation/master/enable-desktops/add-cluster.html		https://osc.github.io/ood-documentation/master/enable-desktops/add-cluster.html
Line 117:		Line 117:

	The correct (yum) installation of python3-websockify puts it in /usr/bin/websockify. Because existing config specs /usr/local/bin, just symlink		The correct (yum) installation of python3-websockify puts it in /usr/bin/websockify. Because existing config specs /usr/local/bin, just symlink

			The final critical step is updating etc/ood/config/ood_portal.yml, adding another entry to the host_regex line (223 last I looked) with the new host name \|newhost.stor'ed in. If this is not done OOD will refuse to match and proxy the vnc viewer back to your browser and NoVNC will not be able to connect.

	=== gnome desktop ===		=== gnome desktop ===

Ekeever1: /* Step -1: NoVNC connection failure debug */

2025-03-21T17:18:51Z

Step -1: NoVNC connection failure debug

← Older revision		Revision as of 17:18, 21 March 2025
Line 9:		Line 9:
	== Step -1: NoVNC connection failure debug ==		== Step -1: NoVNC connection failure debug ==

	If the desktop launches (i.e. squeue shows a sustained job running on the host for more than 10 seconds) but NoVNC fails to connect, this is probably going to be showing up on OOD at		If the desktop launches (i.e. squeue shows a sustained job running on the host for more than 10 seconds) but NoVNC fails to connect, something like this is probably going to be showing up on OOD at
	/var/log/apache2/ood.nic.uoregon.edu_access_ssl.log:		/var/log/apache2/ood.nic.uoregon.edu_access_ssl.log:
	<pre>73.157.215.248 - - [21/Mar/2025:16:24:06 +0000] "GET /rnode/voltar.stor/39352/websockify HTTP/1.1" 404 1212 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"</pre>		<pre>73.157.215.248 - - [21/Mar/2025:16:24:06 +0000] "GET /rnode/voltar.stor/39352/websockify HTTP/1.1" 404 1212 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"</pre>

Ekeever1: /* Step -1: Apparent working user access / bug bypass procedure [UPDATE: new version magically started working as expected] */

2025-03-21T17:18:18Z

Step -1: Apparent working user access / bug bypass procedure [UPDATE: new version magically started working as expected]

← Older revision		Revision as of 17:18, 21 March 2025
Line 7:		Line 7:
	https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.		https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.

	== Step -1: ~~Apparent working user access / bug bypass procedure [UPDATE: new version magically started working as expected]~~ ==		== Step -1: NoVNC connection failure debug ==

	~~Current status: It APPEARS that~~ the ~~OOD~~ desktop ~~launch process, in and of itself, works~~. ~~The browser VNC viewer~~ fails ~~due~~ to ~~problems~~ on ~~ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,~~		If the desktop launches (i.e. squeue shows a sustained job running on the host for more than 10 seconds) but NoVNC fails to connect, this is probably going to be showing up on OOD at
	~~<pre>==> httpd~~/~~access_ssl.~~log ~~<==~~		/var/log/apache2/ood.nic.uoregon.edu_access_ssl.log:
	~~128.223.202.205 - erik-k [08/Jul/2022:16:45:15 -0700] "GET /pun/sys/dashboard/batch_connect/sessions.js?_=1657318165288 HTTP/1.1" 200 5620 "https:~~//ood.nic.uoregon.~~edu/pun/sys/dashboard/batch_connect/sessions" "Mozilla/5~~.~~0 (X11; Linux x86_64; rv~~:~~91.0) Gecko/20100101 Firefox/91.0"~~		<pre>73.157.215.248 - - [21/Mar/2025:16:24:06 +0000] "GET /rnode/voltar.stor/39352/websockify HTTP/1.1" 404 1212 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"</pre>
	~~128~~.~~223~~.~~202~~.~~205~~ - - [08/~~Jul~~/~~2022~~:16:45:~~16 -0700~~] "GET /rnode/~~axis1~~/~~44232~~/websockify HTTP/1.1" 404 ~~196~~ "-" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"</pre>

	~~So clearly something~~ is ~~wrong with websockify setup on~~ the ~~server.~~		This is because the host in question was not added to the rnode_uri's regular expression filter in ood_config.yml as described in host setup below, so OOD is refusing to act as a proxy for this node.

	~~However: The normal login and desktop (~~not ~~desktop-gnome) process succeeds and does launch a VNC viewer on axis1 (''note -~~ the ~~cluster node setup process has only been carried out on axis1 until a procedure for a fully working environment is decrypted~~'~~' Thus this will only work if axis1 is idle, for the moment)~~.

	~~At this point~~, ~~the user may SSH~~ to ~~orthus using~~
	~~<pre>ssh orthus.nic.uoregon.edu -L3456:axis1.stor:590X</pre>~~
	~~where X is determined from the connection.yml that is reachable via OOD by clicking your session ID once it starts. Then connect~~ a ~~VNC viewer to localhost:3456 with the password that is also in the connection.yml file~~.

	== Step 3 - Authentication setup ==		== Step 3 - Authentication setup ==

Ekeever1 at 17:12, 21 March 2025

2025-03-21T17:12:10Z

← Older revision		Revision as of 17:12, 21 March 2025
Line 40:		Line 40:

	per config documents, we need to setup munge/munged and the cluster munge key as well. It appears this system is able to access the cluster scheduler info outputs.		per config documents, we need to setup munge/munged and the cluster munge key as well. It appears this system is able to access the cluster scheduler info outputs.

	~~== Step 5 - desktop configuration ==~~

	~~https://osc.github.io/ood-documentation/master/enable-desktops/add-cluster.html~~

	~~Cluster has been added to desktop environment settings for OOD, now complains about finding slurm. So Slurm must be installed on OOD!~~

	~~Compilation & installation of slurm-21.08.4 on OOD at the same directory as on Orthus.~~

	setting up slurm config on ood:		setting up slurm config on ood:
Line 53:		Line 45:
	* slurm install path must link to ../etc for slurm bins to find config info		* slurm install path must link to ../etc for slurm bins to find config info

	Slurm ~~now installed on OOD. Slurm configuration updated to have OOD in the slurm cluster.~~		Slurm is built at /opt/slurm/version__ as on other systems.

	~~Started mysql on orthus to get slurmdbd working because without it~~ slurm ~~does not work~~ on ~~ood..~~.

	~~Working at https://osc.github.io/ood-documentation/master/enable-desktops/add~~-cluster~~.html ,~~		== Step 5 - cluster node software setup ==

	~~Desktop will now attempt to start however we have an error in~~ the vnc launch command of all things,		Two nonstandard packages are needed on the remote desktop hosts, tigervnc and websockify.

	~~-> It turns out that OOD does not like tigervnc~~		Repo for turbovnc:
	-> ~~Install the turbovnc library and install turbovnc,~~		<pre>[root@voltar yum.repos.d]# cat TurboVNC.repo
	* Contents of /etc/yum.repos.d/TurboVNC.repo:		[TurboVNC]
	~~<pre>~~[TurboVNC]
	name=TurboVNC official RPMs		name=TurboVNC official RPMs
	baseurl=https://sourceforge.net/projects/turbovnc/files		baseurl=https://sourceforge.net/projects/turbovnc/files
Line 72:		Line 61:
	enabled=1</pre>		enabled=1</pre>

	~~Still fails~~ at		Commands:
			<pre>yum install python3-websockify turbovnc
			# The below is due to how ood cluster launcher script on OOD at root@ood3:/etc/ood/config/clusters.d is configured
			# which is in turn because of how websockify ended up installed on the axis nodes so we're default stuck with it now
			cd /usr/local/bin
			ln -s /usr/bin/websockify ./websockify</pre>

			These will install the needed packages to make the thing go.

			== Step 5 - desktop configuration on the cluster ==

			https://osc.github.io/ood-documentation/master/enable-desktops/add-cluster.html

			Desktop will now attempt to start however we have an error in the vnc launch command of all things,

			-> It turns out that OOD does not like tigervnc
			-> Install TurboVNC

			By default it will fail at
	<pre>VNC_OUT=$(vncserver -log "vnc.log" -rfbauth "vnc.passwd" -nohttpd -noxstartup -geometry 800x600 -idletimeout 0 2>&1)</pre> in the job script with		<pre>VNC_OUT=$(vncserver -log "vnc.log" -rfbauth "vnc.passwd" -nohttpd -noxstartup -geometry 800x600 -idletimeout 0 2>&1)</pre> in the job script with
	<pre> Could not start Xvnc.		<pre> Could not start Xvnc.
Line 83:		Line 90:
	opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.23/gems/ood_core-0.18.1/lib/ood_core/batch_connect/templates/vnc.rb:90: VNC_OUT=$(vncserver -log "#{vnc_log}" -rfbauth "#{vnc_passwd}" -nohttpd -noxstartup #{vnc_args} 2>&1)</pre>		opt/ood/ondemand/root/usr/share/gems/2.7/ondemand/2.0.23/gems/ood_core-0.18.1/lib/ood_core/batch_connect/templates/vnc.rb:90: VNC_OUT=$(vncserver -log "#{vnc_log}" -rfbauth "#{vnc_passwd}" -nohttpd -noxstartup #{vnc_args} 2>&1)</pre>

	~~Removing these and will re-try once axis1 is unused (I have only test installed the turbovnc system on axis1)~~.		The invocation now works.

	~~Okay, tested with -nohttpd removed and some progress,~~

	~~Install python3-websockify package.~~

	~~=== websockify ===~~

			Next step at
	https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/modify-cluster-configuration.html		https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/modify-cluster-configuration.html
			does need to be modified.

	This does actually need to be modified. Current content of /etc/ood/config/clusters.d/axis.yml,		This does actually need to be modified. Current content of /etc/ood/config/clusters.d/axis.yml,
Line 105:		Line 108:
	conf: "/opt/slurm/etc/slurm.conf"		conf: "/opt/slurm/etc/slurm.conf"
	batch_connect:		batch_connect:
			basic:
			script_wrapper: \|
			module purge
			%s
			set_host: "host=$(hostname -A \| awk '{print $1}')"
	vnc:		vnc:
	script_wrapper: \|		script_wrapper: \|
Line 110:		Line 118:
	export PATH="/opt/TurboVNC/bin:$PATH"		export PATH="/opt/TurboVNC/bin:$PATH"
	export WEBSOCKIFY_CMD="/usr/local/bin/websockify"		export WEBSOCKIFY_CMD="/usr/local/bin/websockify"
	%s</pre>		%s
			set_host: "host=$(hostname -A \| awk '{print $1}')"</pre>

	This sets the path to turbovnc and websockify.		This sets the path to turbovnc and websockify to make sure they launch and sets the hostname (out of those that can be chosen).

	~~Update:~~ The correct (yum) installation of python3-websockify puts it in /usr/bin. Because existing config specs /usr/local/bin, ~~must~~ symlink		The correct (yum) installation of python3-websockify puts it in /usr/bin/websockify. Because existing config specs /usr/local/bin, just symlink

	=== gnome desktop ===		=== gnome desktop ===
Line 126:		Line 135:
	attributes:		attributes:
	desktop: "gnome"</pre>		desktop: "gnome"</pre>

	~~This gets us far enough that the desktop does actually claim to start and run which is further than we've ever gotten before.~~

Ekeever1: /* Step 3 - Authentication setup */

2025-03-21T17:05:26Z

Step 3 - Authentication setup

← Older revision		Revision as of 17:05, 21 March 2025
Line 25:		Line 25:

	* set ssl cert file and keyfile at ood_config.yml:36		* set ssl cert file and keyfile at ood_config.yml:36
	* under dex:, set ssl: ~~trye~~		* under dex:, set ssl: true
	* utilize most of their given template for LDAP connector, with some minor changes required for our ldap CN/DN/DCs		* utilize most of their given template for LDAP connector, with some minor changes required for our ldap CN/DN/DCs
	* ~~During testing,~~ set OIDCSSLValidateServer Off under httpd conf.d/ood-portal.conf or it will not work		* Must set OIDCSSLValidateServer Off under httpd conf.d/ood-portal.conf after generating or it will not work (self-signed ldap1 cert)

	At this point OOD is online/available and login works ~~but apps are not configured.~~		At this point OOD is online/available and login works

	== Step 4 - cluster configuration ==		== Step 4 - cluster configuration ==

Ekeever1 at 17:04, 21 March 2025

2025-03-21T17:04:33Z

← Older revision		Revision as of 17:04, 21 March 2025
Line 7:		Line 7:
	https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.		https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.

	== Step -1: Apparent working user access / bug bypass procedure ==		== Step -1: Apparent working user access / bug bypass procedure [UPDATE: new version magically started working as expected] ==

	Current status: It APPEARS that the OOD desktop launch process, in and of itself, works. The browser VNC viewer fails due to problems on ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,		Current status: It APPEARS that the OOD desktop launch process, in and of itself, works. The browser VNC viewer fails due to problems on ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,
Line 48:		Line 48:

	Compilation & installation of slurm-21.08.4 on OOD at the same directory as on Orthus.		Compilation & installation of slurm-21.08.4 on OOD at the same directory as on Orthus.

	~~-> slurm config will need to add OOD to config?~~

	setting up slurm config on ood:		setting up slurm config on ood:
	* must have munge-devel installed or it will silently fail to build munge support. One might perhaps speculate that this calls for something more than a tiny, buried not-even-an-error message given how critical munge is to 99% of slurm clusters...		* must have munge-devel installed or it will silently fail to build munge support. One might perhaps speculate that this calls for something more than a tiny, buried not-even-an-error message given how critical munge is to 99% of slurm clusters...
	* slurm install path must link to ../etc for slurm bins to find config info		* slurm install path must link to ../etc for slurm bins to find config info


	Slurm now installed on OOD. Slurm configuration updated to have OOD in the slurm cluster.		Slurm now installed on OOD. Slurm configuration updated to have OOD in the slurm cluster.
Line 90:		Line 87:
	Okay, tested with -nohttpd removed and some progress,		Okay, tested with -nohttpd removed and some progress,

	~~now apparently require~~ websockify ~~to be installed on axis node. For test purposes, installing this manually on axis1.~~		Install python3-websockify package.

	~~https://pypi.org/project/websockify/~~

	~~A better solution is needed for the rest of the nodes..~~.

	=== websockify ===		=== websockify ===
Line 120:		Line 113:

	This sets the path to turbovnc and websockify.		This sets the path to turbovnc and websockify.

			Update: The correct (yum) installation of python3-websockify puts it in /usr/bin. Because existing config specs /usr/local/bin, must symlink

	=== gnome desktop ===		=== gnome desktop ===

Ekeever1 at 23:54, 8 July 2022

2022-07-08T23:54:39Z

← Older revision		Revision as of 23:54, 8 July 2022
Line 16:		Line 16:
	So clearly something is wrong with websockify setup on the server.		So clearly something is wrong with websockify setup on the server.

	However: The normal login and desktop (not desktop-gnome) process succeeds and does launch a VNC viewer on axis1 (``note - the cluster node setup process has only been carried out on axis1 until a procedure for a fully working environment is decrypted``).		However: The normal login and desktop (not desktop-gnome) process succeeds and does launch a VNC viewer on axis1 (''note - the cluster node setup process has only been carried out on axis1 until a procedure for a fully working environment is decrypted'' Thus this will only work if axis1 is idle, for the moment).

	At this point, the user may SSH to orthus using		At this point, the user may SSH to orthus using

Ekeever1 at 23:54, 8 July 2022

2022-07-08T23:54:00Z

← Older revision		Revision as of 23:54, 8 July 2022
Line 7:		Line 7:
	https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.		https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.

	== Step -1: Apparent working user access procedure ==		== Step -1: Apparent working user access / bug bypass procedure ==

	Current status: It APPEARS that the OOD desktop launch process, in and of itself, works. The browser VNC viewer fails due to problems on ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,		Current status: It APPEARS that the OOD desktop launch process, in and of itself, works. The browser VNC viewer fails due to problems on ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,

Ekeever1 at 23:53, 8 July 2022

2022-07-08T23:53:41Z

← Older revision		Revision as of 23:53, 8 July 2022
Line 6:		Line 6:

	https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.		https://leo.leung.xyz/wiki/Open_OnDemand contains a lot of potential underdocumented bugfixes / setup BS.

			== Step -1: Apparent working user access procedure ==

			Current status: It APPEARS that the OOD desktop launch process, in and of itself, works. The browser VNC viewer fails due to problems on ood itself; Any attempt to use the ood vnc viewer provokes the following in the logs,
			<pre>==> httpd/access_ssl.log <==
			128.223.202.205 - erik-k [08/Jul/2022:16:45:15 -0700] "GET /pun/sys/dashboard/batch_connect/sessions.js?_=1657318165288 HTTP/1.1" 200 5620 "https://ood.nic.uoregon.edu/pun/sys/dashboard/batch_connect/sessions" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
			128.223.202.205 - - [08/Jul/2022:16:45:16 -0700] "GET /rnode/axis1/44232/websockify HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"</pre>

			So clearly something is wrong with websockify setup on the server.

			However: The normal login and desktop (not desktop-gnome) process succeeds and does launch a VNC viewer on axis1 (``note - the cluster node setup process has only been carried out on axis1 until a procedure for a fully working environment is decrypted``).

			At this point, the user may SSH to orthus using
			<pre>ssh orthus.nic.uoregon.edu -L3456:axis1.stor:590X</pre>
			where X is determined from the connection.yml that is reachable via OOD by clicking your session ID once it starts. Then connect a VNC viewer to localhost:3456 with the password that is also in the connection.yml file.

	== Step 3 - Authentication setup ==		== Step 3 - Authentication setup ==

Ekeever1 at 05:30, 19 May 2022

2022-05-19T05:30:36Z

← Older revision		Revision as of 05:30, 19 May 2022
Line 105:		Line 105:

	This sets the path to turbovnc and websockify.		This sets the path to turbovnc and websockify.

			=== gnome desktop ===

			Under /etc/ood/config/apps/bc_desktop/ must create a new axis_gnome.yml that sets the default desktop to gnome not mate,

			<pre>title: "Axis \| Gnome Desktop"
			cluster: "axis"
			form:
			- desktop
			attributes:
			desktop: "gnome"</pre>

			This gets us far enough that the desktop does actually claim to start and run which is further than we've ever gotten before.