Service:storage: Difference between revisions
No edit summary |
|||
| Line 9: | Line 9: | ||
Hosts access network storage either over their 10Gbps Ethernet links or 100Gbps IPoIB Infiniband-substrate links. | Hosts access network storage either over their 10Gbps Ethernet links or 100Gbps IPoIB Infiniband-substrate links. | ||
= Main storage: /home/users = | == Main storage: /home/users == | ||
Most systems access NAS on [[Infrastructure:mnemosyne | Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. Vital statistics: | Most systems access NAS on [[Infrastructure:mnemosyne | Mnemosyne]]. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. Vital statistics: | ||
| Line 15: | Line 15: | ||
It is able to maintain up to ~700MBps of write and theoretically up to ~3GBps of read; In the short term, performance may be higher due to caching by nfsd in RAM. | It is able to maintain up to ~700MBps of write and theoretically up to ~3GBps of read; In the short term, performance may be higher due to caching by nfsd in RAM. | ||
= Package tree: /packages = | == Package tree: /packages == | ||
Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1. | Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1. | ||
| Line 23: | Line 23: | ||
The SSDs provide 1.2GBps sequential speed. | The SSDs provide 1.2GBps sequential speed. | ||
= High performance scratch: /storage/warpspeed = | == High performance scratch: /storage/warpspeed == | ||
Mnemosyne hosts a high-performance 16TB scratch volume made from four 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0. This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network. | Mnemosyne hosts a high-performance 16TB scratch volume made from four 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0. This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network. | ||
| Line 29: | Line 29: | ||
Sustained sequential read/write is 1.1GBps over the 10G storage network. Sustained R/W over the Infiniband network has currently reached up to approximately 3.5GBps in the relatively short (~20seconds) term, before thermal limiting reduces throughput to 2GBps. | Sustained sequential read/write is 1.1GBps over the 10G storage network. Sustained R/W over the Infiniband network has currently reached up to approximately 3.5GBps in the relatively short (~20seconds) term, before thermal limiting reduces throughput to 2GBps. | ||
= Upstairs storage = | == Upstairs storage == | ||
Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus. | Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus. | ||
This storage, and package tree, is entirely separate from main storage. | This storage, and package tree, is entirely separate from main storage. | ||
Revision as of 00:02, 29 November 2020
Storage on OACISS systems
OACISS has three separate NAS storage drives, for bulk user data, programs and scratch.
In addition, certain systems may have additional local disks, most often for storage of local images related to container infrastructure like Docker or Shifter.
Hosts access network storage either over their 10Gbps Ethernet links or 100Gbps IPoIB Infiniband-substrate links.
Main storage: /home/users
Most systems access NAS on Mnemosyne. This is the primary storage engine utilizing 14x12TB Seagate Exos disks in RAID-6. It has 135TB of disk available in a single large volume. Vital statistics:
It is able to maintain up to ~700MBps of write and theoretically up to ~3GBps of read; In the short term, performance may be higher due to caching by nfsd in RAM.
Package tree: /packages
Package trees for RH7, RH8 and ppc64le are stored on two 4TB SAS-12G SSDs in RAID-1.
As packages are subject to almost exclusively read access, they are mounted synchronously.
The SSDs provide 1.2GBps sequential speed.
High performance scratch: /storage/warpspeed
Mnemosyne hosts a high-performance 16TB scratch volume made from four 4TB Sabrent Rocket M.2 NVME SSDs configured in RAID-0. This volume is available on the 10Gbit network and on the 100Gbit InfiniBand network.
Sustained sequential read/write is 1.1GBps over the 10G storage network. Sustained R/W over the Infiniband network has currently reached up to approximately 3.5GBps in the relatively short (~20seconds) term, before thermal limiting reduces throughput to 2GBps.
Upstairs storage
Cerberus, Chymera, and the nuc + jetson cluster have access to a smaller 22TB NAS, accessed over 1Gbit ethernet, located on Cerberus.
This storage, and package tree, is entirely separate from main storage.