Service:storage: Difference between revisions

From OACISS Systems Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(18 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Storage on OACISS systems =
[[Category:Service]]


[[Category:Service]]
OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.


OACISS has three separate NAS storage drives, for bulk user data, programs and scratch.
Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.


In addition, certain systems may have additional local disks, most often for storage of images related to container infrastructure like Docker.
== Main storage: /home/users ==


= Main storage =
All systems in the CC utilize the main home directory NAS server on [[Infrastructure:mnemosyne | Mnemosyne]]. This is a primary storage engine with 12 15TB sas-3 SSDs in a raidz2 redundant storage array providing 128TB of disk in one volume.


Most systems access NAS on [[Infrastructure:mnemosyne | Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume.
Single-thread IO tests yield ~1.8GBps sequential read and ~10K IOPS 4K random r/w.


Mnemosyne is connected to the storage network at 40Gbps and has 128GB of memory. It can absorb data at 2.5GBps until RAM runs out, and can sustain roughly 750MBps sequential IO continuously. Any single node is limited to roughly 1GBps of IO by its own 10Gbit link.
== Package tree: /packages ==


For performance reasons, main storage is mounted with -oasync.
Package trees for RH7, RH8 and ppc64le are stored on 15TB SAS-12G SSDs in RAID-1.


= Package tree =
As packages are subject to almost exclusively read access, they are mounted synchronously.


Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1.
== High performance storage: /gpfs/gpfs0 ==


As packages are subject to almost exclusively read access, they are mounted synchronously.
OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.


= High performance scratch =
The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.


Mnemosyne hosts a high-performance 8TB scratch volume made from two 4TB Sabrent Rocket NVME M.2 SSDs.
== BeeGFS storage ==


This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network; Any hosts which have IB access mount it over the IB network.
The majority of nodes have a high performance BeeGFS filesystem available.


The scratch volume is capable of delivering 1.1GBps and 80K IOPS per host over the 10G network (to a limit of about 2.5GBps and 95K IOps total). Over the 100G links, it is capable of delivering an astounding 10GBps and 550K IOPS (sustained sequential write limit ~7GBps).
The data storage lives on a dedicated host, Alexandria, with 15 x 15TB NVME drives arranged in a 180TB raidz2 pool.


= Upstairs storage =
== Upstairs storage ==


Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.
Upstairs storage is a 28TB SSD storage pool living on Sphinx, serving Silicon and the ARM hardware cluster (jetsons, xaviers, orins and OD1K).


This storage, and package tree, is entirely separate from main storage.
This storage, and its package tree, are entirely separate from main storage.

Latest revision as of 01:34, 30 November 2023


OACISS has multiple network attached storage systems for user operations. In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.

Network attached storage operates over the 172.17 Ethernet, which is increasingly being upgraded to 100GbE performance.

Main storage: /home/users

All systems in the CC utilize the main home directory NAS server on Mnemosyne. This is a primary storage engine with 12 15TB sas-3 SSDs in a raidz2 redundant storage array providing 128TB of disk in one volume.

Single-thread IO tests yield ~1.8GBps sequential read and ~10K IOPS 4K random r/w.

Package tree: /packages

Package trees for RH7, RH8 and ppc64le are stored on 15TB SAS-12G SSDs in RAID-1.

As packages are subject to almost exclusively read access, they are mounted synchronously.

High performance storage: /gpfs/gpfs0

OACISS has deployed an IBM GS4S high-performance file server utilizing the Spectrum Scale storage engine. The ESS server cluster has 96 4TB SSDs providing a total of 250TB of storage. The ESS cluster is linked to the private storage network with 200Gbps of Ethernet bandwidth.

The underlying high-performance Spectrum Scale filesystem driver is able to deliver up to approximately 3GBps of sequential IO per thread.

BeeGFS storage

The majority of nodes have a high performance BeeGFS filesystem available.

The data storage lives on a dedicated host, Alexandria, with 15 x 15TB NVME drives arranged in a 180TB raidz2 pool.

Upstairs storage

Upstairs storage is a 28TB SSD storage pool living on Sphinx, serving Silicon and the ARM hardware cluster (jetsons, xaviers, orins and OD1K).

This storage, and its package tree, are entirely separate from main storage.

Retrieved from "https://systems.nic.uoregon.edu/internal-wiki/index.php?title=Service:storage&oldid=3493"