Kestrel System Configuration
Learn about the Kestrel computing system configuration.
Architecture Description
The Kestrel system is a high-performance computing (HPC) system for compute intensive and parallel jobs. User facing login and data analysis and visualization (DAV) nodes will utilize Red Hat Enterprise Linux. Compute nodes will also utilize Red Hat Enterprise Linux. The system utilizes a high-speed, 200 gigabit (Gb)/s Hewlett Packard Enterprise (HPE) Slingshot interconnect. A brief description of the nodes, interconnect, and file systems is provided below.
Compute Node Hardware Details
Kestrel will have 2436 compute nodes available for HPC jobs. Below are the node types on Kestrel:
| Number of Nodes | Processors | Memory | Accelerators | Local Storage |
|---|---|---|---|---|
| 2304 | Dual socket Intel Xeon Sapphire Rapids 52-core processors (104 cores total) | 256 GB DDR5 | N/A | 256 nodes with 1.92 TB NVMe M.2 |
| 132 | Dual socket AMD Genoa 64-core processors (128 cores total) |
384 GB | 4 NVIDIA H100 SXM GPUs, 80 GB Memory |
2 x 1.6 TB NVMe |
| 10 | Dual socket Intel Xeon Sapphire Rapids 52-core processors (104 cores total) | 2 TB DDR5 | N/A | 8 x 1.6 TB NVMe |
| 8 | Dual socket Intel Xeon Sapphire Rapids 52-core processors (104 cores total) | 256 GB DDR5 | 2 NVIDIA A40 GPUs | 2 x 3.84 TB NVMe |
Interconnect
All compute nodes will be connected to the high-performance HPE Slingshot 11 fabric in a dragonfly topology. In the case of the accelerated nodes, each node will have two 200 Gb/s Network Interface Cards (NICs) to the interconnect.
Parallel File System
The Parallel File System (PFS) ProjectFS and ScratchFS on Kestrel is a ClusterStor Lustre file system intended for high-performance I/O. ScratchFS uses a Lustre file system in a hybrid flash-disk configuration providing a total of 27 petabytes (PB) of capacity with 354 gigabytes (GB)/s of IOR bandwidth. ProjectFS will provide 68 PB of capacity with 200 GB/s of IOR bandwidth.
Home File System
The Home File System (HFS) on Kestrel is part of the ClusterStor used for PFS, providing highly reliable storage for user home directories and NREL-specific software. HFS will provide 1.2 PB of capacity. Snapshots of files on the HFS will be available up to 30 days after change/deletion.
Share
