Setup a 1PB File System in Minutes with Amazon FSx for Lustre
Last week, AWS announced that Amazon FSx for Lustre now offers customers the ability to create Hard Disk Drive (HDD)-based file systems and reduce storage costs by up to 80% for throughput-intensive workloads. Those familiar with open source Lustre can appreciate that AWS manages the entire file system for you, and setup of a 1PB file system can be done in just a few minutes.
We looked at the AWS price page, which can be found here, showing FSx now offers a total of eight tiers of the fully managed FSx for Lustre file system; 4 tiers on SSD storage, 2 tiers on HDD storage, and 2 hybrid tiers that include 80% HDD storage with a 20% SSD cache.
Reviewing the release materials and performance data, the key benefit claims by AWS for the FSx for Lustre HDD-based file system can be summarized as:
- Keeps compute cluster utilization high with up to tens of gigabytes per second of throughput
- Responds with sub-ms latencies for metadata file operations by using an SSD-based metadata server
- Accelerates performance by using an optional SSD read cache that provides sub-millisecond latencies and higher IOPS for 20% of file system data
- Simplifies access to S3 data lakes by linking to an S3 bucket and seamlessly presenting S3 objects as files
- Can setup automatic daily backups for business and regulatory compliance
- Priced starting at USD 2.5 cents per GB/month (Ohio Region, prices vary per Region)
According to AWS, the customers that could most benefit seem to be the ones running throughput-intensive workloads that don’t require the sub-millisecond latencies of SSD storage, including genome analysis, financial simulations, seismic data processing and video transcoding. We also confirmed that all of the HDD tiers, with and without the 20% SSD cache, use SSD for their Lustre metadata server operations. For more detailed information, see these links below:
AWS Blog: High-Performance HDD Storage for Amazon FSx for Lustre File Systems