Service:storage
OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.
Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.
Main storage: /home/users
Most systems access NAS on Mnemosyne. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume.
Mnemosyne has 512GB of memory for caching and can sustain up to 700MBps sequential write and ~3GBps sequential read.
Package tree: /packages
Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1.
As packages are subject to almost exclusively read access, they are mounted synchronously.
The SSDs provide 1.2GBps sequential speed.
High performance storage: /gpfs/gpfs0
OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.
The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.
High performance scratch: /storage/warpspeed
Mnemosyne hosts a high-performance 16TB scratch volume made from four 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0. This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network.
Sustained sequential read/write is 1.1GBps over the 10G storage network. Sustained R/W over the Infiniband network has currently reached up to approximately 3.5GBps in the relatively short (~20seconds) term, before thermal limiting reduces throughput to 2GBps.
Upstairs storage
Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.
This storage, and package tree, is entirely separate from main storage.