Memory Machine Cloud
Solutions
Supercharge Pipeline Performance HPC Cost Optimization Made Easy
“ MMCloud has helped our team in the Computational Biology Core at MDI Biological Laboratory better utilize AWS Spot EC2 instances to save cloud computing costs. In addition, the WaveWatcher tool has let us observe resource usage in real time, providing new insights into optimizing resource allocations for our work with Nextflow pipelines.”
– Joel Graber, Director of the Computational Biology and Bioinformatics Core
Memory Machine™ Cloud
Challenges and Solution
- Learning a complex and new infrastructure management environment (AWS, GCP, etc.)
- Fine-tuning pipeline performance for the ideal virtual machine type
- Achieving cost-effectiveness through Spot instance usage
- Ensuring scalable automation across pipelines, resources, and users
Memory Machine Cloud’s Nextflow plugin “nf-float” enables users to easily automate and provision cloud computing resources to supercharge pipeline execution, safely and reliably run on low-cost Spot instances, and optimize batch job execution. Key features like SpotSurfer for job recovery, WaveRider for real-time optimization, and WaveWatcher for resource insight make bioinformatics in the cloud a breeze.
The nf-float plugin unlocks the added value of Memory Machine Cloud
Memory Machine Cloud is a feature-rich cloud batch executor, easily deployed with a few lines in the Nextflow configuration file. The benefits include cost savings, improved time-to-results, and deep insight into resource utilization at the container level.
WaveWatcher Observability Service
When Memory Machine launches a worker node, it opens a communication channel to a resource monitor inside the container. Real-time metrics are available as a CSV file or can be viewed graphically. The WaveWatcher Observability Service displays detailed cloud costs for each job in real-time, and provides reports on cloud cost for each app, each user, and each group on the GUI. It also provides fine-grained data on CPU / Memory / Network / Storage I/O utilizations, as well as the carbon footprint. Use this data for optimizing cloud resources for your workloads.
To pinpoint opportunities for optimization, WaveWatcher displays real time app usage of CPU, memory, network, and storage.
The screen on the left shows the WaveRider could lower costs by starting the workload on a smaller instance, them moving to a larger instance only when more resources are needed.
WaveRider Continuous Optimization Service
Transparent to NextFlow, WaveRider provides the ability to migrate running jobs to optimally-sized virtual machines based on their real-time resource utilization. The WaveRider service enables users to optimize cost and wall-clock time with a few mouse clicks.
The grey areas show memory and CPU usage on different compute instances as WaveRider continuously right sizes based on the need for resources during runtime.
SpotSurfer Checkpointing and Restore Service
Spot instances are a way for cloud service providers to monetize idle compute capacity. A Spot instance may be offered at a discount of up to 90% off the price of an On-demand instance. There is a catch – a Spot instance may be reclaimed by the cloud service providers with a typical warning of only two minutes. Memory Machine Cloud includes the SpotSurfer checkpointing and restore service that allows a running job to move seamlessly from a Spot instance flagged for reclamation to a new Spot or on-demand instance. This feature allows users to reduce cloud costs significantly without increasing wall clock time.
The grey areas show memory and CPU usage during runtime. The white areas represent the time period when the workload is automatically migrated to a new compute instance and restarted.