Service:storage: Difference between revisions
No edit summary |
|||
| Line 3: | Line 3: | ||
[[Category:Service]] | [[Category:Service]] | ||
OACISS | OACISS has three separate NAS storage drives, for bulk user data, programs and scratch. | ||
In addition, certain systems may have additional local disks, most often for storage of images related to container infrastructure like Docker. | In addition, certain systems may have additional local disks, most often for storage of images related to container infrastructure like Docker. | ||
= Main storage = | |||
Most systems access NAS on [[Infrastructure:mnemosyne | Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. | Most systems access NAS on [[Infrastructure:mnemosyne | Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. | ||
Mnemosyne is connected to the storage network at 40Gbps and has 128GB of memory. It can absorb data at 2.5GBps until | Mnemosyne is connected to the storage network at 40Gbps and has 128GB of memory. It can absorb data at 2.5GBps until RAM runs out, and can sustain roughly 750MBps sequential IO continuously. Any single node is limited to roughly 1GBps of IO by its own 10Gbit link. | ||
For performance reasons, main storage is mounted with -oasync. | |||
= | = Package tree = | ||
Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1. | |||
== | As packages are subject to almost exclusively read access, they are mounted synchronously. | ||
= High performance scratch = | |||
Mnemosyne hosts a high-performance 8TB scratch volume made from two 4TB Sabrent Rocket NVME M.2 SSDs. | |||
This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network; Any hosts which have IB access mount it over the IB network. | |||
The scratch volume is capable of delivering 1.1GBps and 80K IOPS per host over the 10G network (to a limit of about 2.5GBps and 95K IOps total). Over the 100G links, it is capable of delivering an astounding 10GBps and 550K IOPS (sustained sequential write limit ~7GBps). | |||
= Upstairs storage = | |||
Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus. | Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus. | ||
This storage, and package tree, is entirely separate from main storage. | This storage, and package tree, is entirely separate from main storage. | ||
Revision as of 22:44, 3 October 2020
Storage on OACISS systems
OACISS has three separate NAS storage drives, for bulk user data, programs and scratch.
In addition, certain systems may have additional local disks, most often for storage of images related to container infrastructure like Docker.
Main storage
Most systems access NAS on Mnemosyne. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume.
Mnemosyne is connected to the storage network at 40Gbps and has 128GB of memory. It can absorb data at 2.5GBps until RAM runs out, and can sustain roughly 750MBps sequential IO continuously. Any single node is limited to roughly 1GBps of IO by its own 10Gbit link.
For performance reasons, main storage is mounted with -oasync.
Package tree
Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1.
As packages are subject to almost exclusively read access, they are mounted synchronously.
High performance scratch
Mnemosyne hosts a high-performance 8TB scratch volume made from two 4TB Sabrent Rocket NVME M.2 SSDs.
This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network; Any hosts which have IB access mount it over the IB network.
The scratch volume is capable of delivering 1.1GBps and 80K IOPS per host over the 10G network (to a limit of about 2.5GBps and 95K IOps total). Over the 100G links, it is capable of delivering an astounding 10GBps and 550K IOPS (sustained sequential write limit ~7GBps).
Upstairs storage
Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.
This storage, and package tree, is entirely separate from main storage.