Service:storage - Revision history

Nic-wiki at 01:34, 30 November 2023

2023-11-30T01:34:21Z

@@ Line 22: / Line 22: @@
 The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.
 == BeeGFS storage ==
-Experimental deployment of a BeeGFS high-performance NFS is underway.
+The majority of nodes have a high performance BeeGFS filesystem available.
 == Upstairs storage ==
-Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.
+Upstairs storage is a 28TB SSD storage pool living on Sphinx, serving Silicon and the ARM hardware cluster (jetsons, xaviers, orins and OD1K).
-This storage, and package tree, is entirely separate from main storage.
+This storage, and its package tree, are entirely separate from main storage.

Nic-wiki: /* Package tree: /packages */

2023-11-30T01:19:04Z

Package tree: /packages

← Older revision		Revision as of 01:19, 30 November 2023
Line 13:		Line 13:
	== Package tree: /packages ==		== Package tree: /packages ==

	Package trees for RH7, RH8 and ppc64le are stored on ~~two 4TB~~ SAS-12G SSDs in RAID-1.		Package trees for RH7, RH8 and ppc64le are stored on 15TB SAS-12G SSDs in RAID-1.

	As packages are subject to almost exclusively read access, they are mounted synchronously.		As packages are subject to almost exclusively read access, they are mounted synchronously.

	~~The SSDs provide 1.2GBps sequential speed.~~

	== High performance storage: /gpfs/gpfs0 ==		== High performance storage: /gpfs/gpfs0 ==

Nic-wiki: /* Main storage: /home/users */

2023-11-30T01:18:42Z

Main storage: /home/users

← Older revision		Revision as of 01:18, 30 November 2023
Line 7:		Line 7:
	== Main storage: /home/users ==		== Main storage: /home/users ==

	~~Most~~ systems ~~access~~ NAS on [[Infrastructure:mnemosyne \| Mnemosyne]]. This is ~~the~~ primary storage engine ~~utilizing 14x12TB Seagate Exos disks~~ in ~~mdadm RAID-6. It has 135TB~~ of disk ~~available~~ in ~~a single large~~ volume.		All systems in the CC utilize the main home directory NAS server on [[Infrastructure:mnemosyne \| Mnemosyne]]. This is a primary storage engine with 12 15TB sas-3 SSDs in a raidz2 redundant storage array providing 128TB of disk in one volume.

	~~Mnemosyne has 512GB of memory for caching and can sustain up to 700MBps~~ sequential ~~write~~ and ~~~3GBps sequential read~~.		Single-thread IO tests yield ~1.8GBps sequential read and ~10K IOPS 4K random r/w.

	== Package tree: /packages ==		== Package tree: /packages ==

Ekeever1 at 06:27, 16 May 2022

2022-05-16T06:27:47Z

← Older revision		Revision as of 06:27, 16 May 2022
Line 69:		Line 69:

	* Large block 75R/25W mixed IO:		* Large block 75R/25W mixed IO:
	** 1 thread:		** 1 thread: 889/297MBps
	** 2 threads:		** 2 threads: 1567/525MBps
	** 4 threads:		** 4 threads: 2620/879MBps
	* Medium block 75R/25W mixed IO:		* Medium block 75R/25W mixed IO:
	** 1 thread: 449/151MBps		** 1 thread: 449/151MBps

Ekeever1 at 06:23, 16 May 2022

2022-05-16T06:23:56Z

@@ Line 7: / Line 7: @@
 == Main storage: /home/users ==
-Most systems access NAS on [[Infrastructure:mnemosyne | Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume.
+Most systems access NAS on [[Infrastructure:mnemosyne | Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in mdadm RAID-6. It has 135TB of disk available in a single large volume.
 Mnemosyne has 512GB of memory for caching and can sustain up to 700MBps sequential write and ~3GBps sequential read.
@@ Line 38: / Line 38: @@
 * 12 x 18T disks forming a 150TB volume in RaidZ2
-BeeGFS itself has been assigned a directory on the warpspeed filesystem for its own metadata.
+BeeGFS itself has been temporarily assigned a directory on the warpspeed filesystem (nvme mdadm raid1) for its own metadata.
 == Upstairs storage ==

Ekeever1 at 04:51, 14 May 2022

2022-05-14T04:51:50Z

← Older revision		Revision as of 04:51, 14 May 2022
Line 27:		Line 27:
	== High performance scratch: /storage/warpspeed ==		== High performance scratch: /storage/warpspeed ==

	Mnemosyne hosts a ~~high~~-~~performance 16TB~~ scratch volume made from ~~four~~ 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0~~. This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network~~.		Mnemosyne hosts a low-latency 8TB scratch volume made from two 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0.

	~~Sustained sequential read/write~~ is 1.~~1GBps over the 10G storage network. Sustained R/W over the Infiniband network has currently reached up~~ to ~~approximately 3.5GBps~~ in the ~~relatively short (~20seconds) term, before thermal limiting reduces throughput to 2GBps~~.		== BeeGFS storage ==

			Experimental deployment of a BeeGFS high-performance NFS is underway.

			The backing store is a ZFS pool to which are assigned:
			* 2 mirrored 4T NVME drives for ZFS metadata
			* 2 4T NVMEs assigned as cache
			* 12 x 18T disks forming a 150TB volume in RaidZ2

			BeeGFS itself has been assigned a directory on the warpspeed filesystem for its own metadata.

	== Upstairs storage ==		== Upstairs storage ==

Nic-wiki: /* Storage on OACISS systems */

2021-07-07T03:32:02Z

Storage on OACISS systems

← Older revision		Revision as of 03:32, 7 July 2021
Line 1:		Line 1:
	~~= Storage on OACISS systems =~~

	[[Category:Service]]		[[Category:Service]]

	OACISS has ~~three separate NAS~~ storage ~~drives,~~ for ~~bulk~~ user ~~data, programs and scratch~~.		OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.

	In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.

	~~Hosts access network~~ storage ~~either~~ over ~~their 10Gbps~~ Ethernet ~~links or 100Gbps IPoIB Infiniband-substrate links~~.		Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.

	== Main storage: /home/users ==		== Main storage: /home/users ==

Nic-wiki at 03:25, 7 July 2021

2021-07-07T03:25:22Z

← Older revision		Revision as of 03:25, 7 July 2021
Line 22:		Line 22:

	The SSDs provide 1.2GBps sequential speed.		The SSDs provide 1.2GBps sequential speed.

			== High performance storage: /gpfs/gpfs0 ==

			OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.

			The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.

	== High performance scratch: /storage/warpspeed ==		== High performance scratch: /storage/warpspeed ==

Nic-wiki: /* Main storage: /home/users */

2021-07-07T03:21:05Z

Main storage: /home/users

← Older revision		Revision as of 03:21, 7 July 2021
Line 11:		Line 11:
	== Main storage: /home/users ==		== Main storage: /home/users ==

	Most systems access NAS on [[Infrastructure:mnemosyne \| Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. ~~Vital statistics:~~		Most systems access NAS on [[Infrastructure:mnemosyne \| Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume.

	~~It is able to maintain~~ up to ~700MBps of write and ~~theoretically up to~~ ~3GBps of read~~; In the short term, performance may be higher due to caching by nfsd in RAM~~.		Mnemosyne has 512GB of memory for caching and can sustain up to 700MBps sequential write and ~3GBps sequential read.

	== Package tree: /packages ==		== Package tree: /packages ==

Nic-wiki at 00:02, 29 November 2020

2020-11-29T00:02:18Z

← Older revision		Revision as of 00:02, 29 November 2020
Line 9:		Line 9:
	Hosts access network storage either over their 10Gbps Ethernet links or 100Gbps IPoIB Infiniband-substrate links.		Hosts access network storage either over their 10Gbps Ethernet links or 100Gbps IPoIB Infiniband-substrate links.

	= Main storage: /home/users =		== Main storage: /home/users ==

	Most systems access NAS on [[Infrastructure:mnemosyne \| Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. Vital statistics:		Most systems access NAS on [[Infrastructure:mnemosyne \| Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. Vital statistics:
Line 15:		Line 15:
	It is able to maintain up to ~700MBps of write and theoretically up to ~3GBps of read; In the short term, performance may be higher due to caching by nfsd in RAM.		It is able to maintain up to ~700MBps of write and theoretically up to ~3GBps of read; In the short term, performance may be higher due to caching by nfsd in RAM.

	= Package tree: /packages =		== Package tree: /packages ==

	Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1.		Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1.
Line 23:		Line 23:
	The SSDs provide 1.2GBps sequential speed.		The SSDs provide 1.2GBps sequential speed.

	= High performance scratch: /storage/warpspeed =		== High performance scratch: /storage/warpspeed ==

	Mnemosyne hosts a high-performance 16TB scratch volume made from four 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0. This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network.		Mnemosyne hosts a high-performance 16TB scratch volume made from four 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0. This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network.
Line 29:		Line 29:
	Sustained sequential read/write is 1.1GBps over the 10G storage network. Sustained R/W over the Infiniband network has currently reached up to approximately 3.5GBps in the relatively short (~20seconds) term, before thermal limiting reduces throughput to 2GBps.		Sustained sequential read/write is 1.1GBps over the 10G storage network. Sustained R/W over the Infiniband network has currently reached up to approximately 3.5GBps in the relatively short (~20seconds) term, before thermal limiting reduces throughput to 2GBps.

	= Upstairs storage =		== Upstairs storage ==

	Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.		Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.

	This storage, and package tree, is entirely separate from main storage.		This storage, and package tree, is entirely separate from main storage.