Service:storage

OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.

Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.

Main storage: /home/users

All systems in the CC utilize the main home directory NAS server on Mnemosyne. This is a primary storage engine with 12 15TB sas-3 SSDs in a raidz2 redundant storage array providing 128TB of disk in one volume.

Single-thread IO tests yield ~1.8GBps sequential read and ~10K IOPS 4K random r/w.

Package tree: /packages

Package trees for RH7, RH8 and ppc64le are stored on 15TB SAS-12G SSDs in RAID-1.

As packages are subject to almost exclusively read access, they are mounted synchronously.

High performance storage: /gpfs/gpfs0

OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.

The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.

High performance scratch: /storage/warpspeed

Mnemosyne hosts a low-latency 8TB scratch volume made from two 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0.

BeeGFS storage

Experimental deployment of a BeeGFS high-performance NFS is underway.

The backing store is a ZFS pool to which are assigned:

• 2 mirrored 4T NVME drives for ZFS metadata
• 2 4T NVMEs assigned as cache
• 12 x 18T disks forming a 150TB volume in RaidZ2

BeeGFS itself has been temporarily assigned a directory on the warpspeed filesystem (nvme mdadm raid1) for its own metadata.

Preliminary benchmark results using fio (posixaio, size=4G, end_fsync=1, iodepth=16, 60sec results) are below. 1M = 2^20.

• Large block (1M) sequential read
  • 1 thread: 1520MBps
  • 2 threads: 2260MBps
  • 4 threads: 4835MBps
  • 8 threads: 9017MBps
• Large block random read
  • 1 thread: 1020MBps
  • 2 threads: 2020MBps
  • 4 threads: 3790MBps
• Medium block (64K) sequential read
  • 1 thread: 1220MBps
  • 2 threads: 2141MBps
  • 4 threads: 4508MBps
• Medium block random read
  • 1 thread: 370MBps
  • 2 threads: 883MBps
  • 4 threads: 1735MBps
• Small block (4K) sequential read
  • 1 thread:
  • 2 threads:
  • 4 threads:
• Small block random read
  • 1 thread: 42MBps
  • 2 threads:
  • 4 threads:

• Large block 75R/25W mixed IO:
  • 1 thread: 889/297MBps
  • 2 threads: 1567/525MBps
  • 4 threads: 2620/879MBps
• Medium block 75R/25W mixed IO:
  • 1 thread: 449/151MBps
  • 2 threads:
  • 4 threads:
• Small block 75R/25W mixed IO:
  • 1 thread:
  • 2 threads:
  • 4 threads:

Upstairs storage

Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.

This storage, and package tree, is entirely separate from main storage.