FSx for Lustre is a distributed parallel filesystem that’s used as part of HPC (high-performance computing) workloads across a variety of complex fields (including life sciences, machine learning, meteorology, finance, and media). It is AWS’s managed version of the Lustre filesystem, and while it’s a popular high-performance storage option, it can quickly become unaffordable as HPC projects scale.
This article directly compares the cost and performance for FSx for Lustre and its alternatives, as well as discussing their scalability and compatibility with your existing HPC workloads.
cunoFS is a high-performance alternative to Amazon’s FSx for Lustre filesystem
This article compares FSx for Lustre, AWS Mountpoint, and cunoFS. cunoFS allows HPC users to use Amazon’s S3 object storage as a high-performance POSIX-compatible filesystem that outperforms FSx for Lustre on both cost and performance, as demonstrated in the benchmarks on this page.
cunoFS allows you to run your existing applications and scripts (Python, C++, bash, etc.) as normal, which massively improves your development efficiency.
Features and compatibility: comparing FSx for Lustre, AWS Mountpoint, and cunoFS
FSx for Lustre vs. Mountpoint for Amazon S3
Mountpoint for Amazon S3 allows you to mount an Amazon S3 bucket as a local filesystem, but it has very limited POSIX compatibility and does not support many common filesystem operations. While Mountpoint is free to use and you only pay for your S3 storage costs, the downside is that it doesn’t work for anything beyond listing files, reading them, or creating them.
Modifying and renaming files, deleting directories, and using symlinks and file locking are all beyond Mountpoint’s capabilities, causing the majority of HPC workloads that try to use it to fail.
FSx for Lustre with S3 integration
FSx for Lustre has an integration with S3 that allows you to sync your FSx for Lustre files with S3 objects. You can set this up by adding a data repository association (a link to your S3 bucket) when creating your FSx for Lustre filesystem.
This allows you to temporarily import all your data from S3, run your workload on the POSIX-compatible FSx for Lustre filesystem, and then shut down the FSx for Lustre filesystem and export your entire output dataset back to S3.
Many choose FSX for Lustre with S3 integration when they need better performance than S3 offers natively but do not want to bear the full costs of permanently storing data in FSX for Lustre. Using this will give the same performance as FSx for Lustre does; however, it’s still nowhere near as cheap as running all your workloads on S3.
FSx for Lustre vs. cunoFS
Whether you use the standard FSx for Lustre, or FSx for Lustre with S3 integration, it’s still much more expensive than using cunoFS, which runs all your workloads directly on S3.
S3 is cheaper than FSx for Lustre, with faster performance, and the ability to scale to larger sizes of workload. However, its weakness has always been its lack of POSIX compatibility. Having to rewrite all your code so that every filesystem call works with S3’s API is tedious and time-consuming.
cunoFS allows you to get around S3’s limitations. Your workloads can continue to make POSIX calls while also using S3, as cunoFS automatically intercepts all filesystem calls and replaces them with the correct S3 API calls.
Performance benchmarks: cunoFS vs. FSx for Lustre
The table below compares the cost vs. performance (in seconds) of FSx for Lustre and cunoFS. We provide this data based on one copy operation per day for a month and on ten copy operations per day for a month, so you can see how each solution scales.
Metrics based on one copy operation per day for a month
| Read | Write | ||||||||
| Copy 5 x 32 GiB into RAM disk | Copy Linux kernel source into RAM disk | Copy 5 x 32 GiB into storage system | Copy Linux kernel source into storage system | ||||||
| FSx for Lustre (all options) |
Duration (seconds) |
278-525s | 224-292s | 287-1046s | 145-186s | ||||
|
Monthly cost ($) |
$150-644 | ||||||||
| FSx for Lustre
SSD Scratch |
Duration (seconds) |
525s | 245s | 1046s | 182s | ||||
|
Monthly cost ($) |
$150 | ||||||||
| FSx for Lustre
SSD 125 MBps |
Duration (seconds) |
327s | 224s | 330s | 149s | ||||
|
Monthly cost ($) |
$178 | ||||||||
| FSx for Lustre
SSD 250 MBps Duration is interpolated. |
Duration (seconds) |
320s | 228s | 324s | 153s | ||||
|
Monthly cost ($) |
$225 | ||||||||
| FSx for Lustre
SSD 500 MBps Duration is interpolated. |
Duration (seconds) |
306s | 234s | 312s | 162s | ||||
|
Monthly cost ($) |
$365 | ||||||||
| FSx for Lustre
SSD 1000 MBps |
Duration (seconds) |
278s | 248s | 287s | 178s | ||||
|
Monthly cost ($) |
$644 | ||||||||
| FSx for Lustre
HDD 12 MBps |
Duration (seconds) |
375s | 245s | 439s | 186s | ||||
|
Monthly cost ($) |
$154 | ||||||||
| FSx for Lustre
HDD 12 MBps with SSD cache |
Duration (seconds) |
309s | 251s | 370s | 150s | ||||
|
Monthly cost ($) |
$252 | ||||||||
| FSx for Lustre
HDD 40 MBps |
Duration (seconds) |
382s | 234s | 420s | 145s | ||||
|
Monthly cost ($) |
$153 | ||||||||
| FSx for Lustre
HDD 40 MBps with SSD cache |
Duration (seconds) |
293s | 216s | 377s | 130s | ||||
|
Monthly cost ($) |
$182 | ||||||||
| cunoFS with S3 *
*Does not include license fee for commercial use |
Duration (seconds) |
29s | 18s | 37s | 79s | ||||
|
Monthly cost ($) |
$25 | $26 | $26 | $41 | |||||
The data in this table clearly demonstrates that cunoFS is cheaper and faster than FSx for Lustre when reading and writing data.
Metrics based on ten copy operations per day for a month
This data is based on estimates that were extrapolated from the costs of the single copy per day data, taking into account the cost per GB of each system, and burst credits where relevant. It also assumes that the storage system is idle for the full 24 hours between each run.
| Read | Write | ||||||||
| Copy 5 x 32 GiB into RAM disk | Copy Linux kernel source into RAM disk | Copy 5 x 32 GiB into storage system | Copy Linux kernel source into storage system | ||||||
| FSx for Lustre (all options) |
Duration (seconds) |
2776-5252s | 2242-2925s | 2873-10461s | 1445-3769s | ||||
|
Monthly cost ($) |
$150-644 | ||||||||
| FSx for Lustre SSD Scratch |
Duration (seconds) |
5253s | 2448s | 10461s | 1822s | ||||
|
Monthly cost ($) |
$150 | ||||||||
| FSx for Lustre SSD 125 MBps |
Duration (seconds) |
3266s | 2242s | 3298s | 1489s | ||||
|
Monthly cost ($) |
$178 | ||||||||
| FSx for Lustre SSD 250 MBps Duration is interpolated. |
Duration (seconds) |
3196s | 2275s | 3237s | 1531s | ||||
|
Monthly cost ($) |
$225 | ||||||||
| FSx for Lustre SSD 500 MBps Duration is interpolated. |
Duration (seconds) |
3056s | 2342s | 31165s | 1616s | ||||
|
Monthly cost ($) |
$365 | ||||||||
| FSx for Lustre SSD 1000 MBps |
Duration (seconds) |
2776s | 2477s | 2873s | 1784s | ||||
|
Monthly cost ($) |
$644 | ||||||||
| FSx for Lustre HDD 12 MBps |
Duration (seconds) |
3750s | 2450s | 4388s | 1863s | ||||
|
Monthly cost ($) |
$154 | ||||||||
| FSx for Lustre HDD 12 MBps with SSD cache |
Duration (seconds) |
3088s | 2513s | 3696s | 1503s | ||||
|
Monthly cost ($) |
$252 | ||||||||
| FSx for Lustre HDD 40 MBps |
Duration (seconds) |
3820s | 2392s | 4198s | 1445s | ||||
|
Monthly cost ($) |
$153 | ||||||||
| FSx for Lustre HDD 40 MBps with SSD cache |
Duration (seconds) |
2925s | 2163s | 3769s | 1298s | ||||
|
Monthly cost ($) |
$182 | ||||||||
| cunoFS with S3*
*Does not include license fee for commercial use |
Duration (seconds) |
288s | 175s | 373s | 792s | ||||
|
Monthly cost ($) |
$27 | $34 | $35 | $191 | |||||
| Read | Write | ||||||||
| Copy 5 x 32 GiB into RAM disk | Copy Linux kernel source into RAM disk | Copy 5 x 32 GiB into storage system | Copy Linux kernel source into storage system | ||||||
| FSx for Lustre SSD Scratch |
Duration (seconds) |
5253s | 2448s | 10461s | 1822s | ||||
|
Monthly cost ($) |
$150 | ||||||||
| FSx for Lustre SSD 125 MBps |
Duration (seconds) |
3266s | 2242s | 3298s | 1489s | ||||
|
Monthly cost ($) |
$178 | ||||||||
| FSx for Lustre SSD 250 MBps Duration is interpolated. |
Duration (seconds) |
3196s | 2275s | 3237s | 1531s | ||||
|
Monthly cost ($) |
$225 | ||||||||
| FSx for Lustre SSD 500 MBps Duration is interpolated. |
Duration (seconds) |
3056s | 2342s | 31165s | 1616s | ||||
|
Monthly cost ($) |
$365 | ||||||||
| FSx for Lustre SSD 1000 MBps |
Duration (seconds) |
2776s | 2477s | 2873s | 1784s | ||||
|
Monthly cost ($) |
$644 | ||||||||
| FSx for Lustre HDD 12 MBps |
Duration (seconds) |
3750s | 2450s | 4388s | 1863s | ||||
|
Monthly cost ($) |
$154 | ||||||||
| FSx for Lustre HDD 12 MBps with SSD cache |
Duration (seconds) |
3088s | 2513s | 3696s | 1503s | ||||
|
Monthly cost ($) |
$252 | ||||||||
| FSx for Lustre HDD 40 MBps |
Duration (seconds) |
3820s | 2392s | 4198s | 1445s | ||||
|
Monthly cost ($) |
$153 | ||||||||
| FSx for Lustre HDD 40 MBps with SSD cache |
Duration (seconds) |
2925s | 2163s | 3769s | 1298s | ||||
|
Monthly cost ($) |
$182 | ||||||||
| cunoFS with S3*
*Does not include license fee for commercial use |
Duration (seconds) |
288s | 175s | 373s | 792s | ||||
|
Monthly cost ($) |
$27 | $34 | $35 | $191 | |||||
You can also see our benchmarks that show how cunoFS outpaces previous solutions like s3fs, goofys, and s3ql.
How to switch from FSx for Lustre to cunoFS
Setting up cunoFS to replace FSx for Lustre takes only a few steps. It’s easy to do, and worthwhile as it’s the best available alternative to FSx for Lustre.
You’ll need to create an S3 bucket and IAM credentials to access it, and then download and install cunoFS. Once it’s connected to your bucket, you can start using it immediately in either Direct Intercept mode (for fast CLI filesystem access) or in our high performance Fusion or high-compatibility Flexmount modes.
Next, copy your data from FSx for Lustre to S3 using cunoFS on the command line, and run your unmodified workloads on it. We recommend using Fusion mode for the highest performance, which combines EFS with S3 via cunoFS for the fastest possible performance, without the increased costs of a pure-EFS deployment.
The advantages of cunoFS go beyond cost and performance: we don’t scramble files, and we don’t lock you into using our product or a single vendor’s object storage. cunoFS is free for personal use and evaluation, and business users can contact us to discuss our commercially supported plans. We’re also keen to hear about how we can extend cunoFS to meet your needs, so if you have any unique requirements, please don’t hesitate to book a meeting.
