Accelerating Biotech Breakthroughs with Cloud HPC Solution on Cloudam for a Better Future of Human Health.
Technological innovations over the past decade have significantly reduced the cost of human genome sequencing, enabling researchers to conduct more in-depth studies of various diseases at a lower cost. Generally, these studies include sequencing large numbers of samples to find the causes of genetic disorders and diseases.
Image Source: NIH
The rapid evolution of genome sequencing technologies, from MERFISH (Multiplexed Error Robust Fluorescence In Situ Hybridization) to Slide-seq, and then to the spatial transcriptomics Stereo-seq, brings an evergrowing spatial resolution and coverage of the transcriptome. Among them, Stereo-seq, a spatial transcriptomics technology independently developed by BGI in early 2021, has achieved "nano-scale resolution" and "centimeter-scale panoramic field of view" at the same time. These advancements also bring challenges to HPC systems.
Early Cooperation between BGI & Cloudam
The earliest cooperation between BGI and Cloudam HPC started from the end-to-end HPC-as-a-service platform on Cloudam which allows researchers in BGI's spatiotemporal transcriptomics project to quickly transfer data, load software, and schedule resources on the platform. The workflow orchestration system and hardware scheduling system optimized with AI have significantly boosted their work efficiency. There was no need for researchers to learn UNIX basics and HPC scheduler commands anymore with these 2 systems. Researchers could fully concentrate on sequencing analysis without worries about queuing for resources. Through the project, BGI highly praised the convenience of life sciences computing brought by Cloudam, therefore has deepened its cooperation with Cloudam to the next level.
Customized Cloud-HPC Solution for BGI by Cloudam
Due to the high demands on resources and prices in the genetic analysis industry, a highly flexible, scalable and automated intelligent HPC system has become the key to the success of the cooperation. BGI put forward three requirements: in job operation, the system must be able to integrate scheduling and orchestration tools commonly used in biotechnology, support software containers, and schedule SPOT computing clusters flexibly; in terms of deployment, operation, and maintenance, it should be able to quickly build and manage computing and storage environments on the cloud, easing the workload for the IT team; in terms of budget and cost control, it needs a cost-effective solution considering multiple dimensions such as computing and storage. Based on the 3 requirements, Cloudam partnered with AWS to deploy a cloud-based HPC system specifically designed for BGI’s team in Latvia. BGI used a sequencer to generate sequencing data locally and pushed the data to the cloud through a dedicated line. Cloudam used the scheduling system to flexibly apply for SPOT computing resources for researchers, which are automatically released after the computing is over. During this process, Cloudam’s scheduling system provided an automatic downgrade mechanism for instance types and queues to ensure high availability of resources, and make sure that researchers can always complete tasks on time. After the computation, the results were automatically uploaded to S3 for BGI customers to download for later usage.
In the meantime, Cloudam relentlessly optimized the solution from multiple dimensions to control the sequencing cost. In terms of computation, Cloudam supported flexible scheduling of SPOT instances, which at most can save up to 90% of machine time costs compared to traditional On-Demand instances. Storage-wise, according to the features of BGI's data volume that varies greatly depending on the size of the project, the platform smartly selected the file system according to the needs, avoiding extra costs due to a large amount of fragmented idle space. About the process, the professional pre/post-processing scripts by Cloudam enabled an automatic flow of data between object storage and NFS. Data would only fall on NFS during computing, thus saving overheads on storage. Meanwhile, users could use the storage strategies on Cloudam to automatically archive data which further saves costs. In addition to qualifiable expenses, other costs were also decreasing, including IT operations and labor costs. The HPC system hosted by Cloudam provided transparency of sophisticated configuration work to IT personnel. Cloudam was also responsible for the audition of data availability, security, and compliance.
Results
Cloudam’s assistance in HPC solutions enabled BGI researchers to have access to unlimited cloud computing resources, which greatly improves the speed and scale of genetic analysis. On the one hand, the needs of commercial projects were met; on the other hand, the data needed for the research papers were obtained. The cutting-edge technologies and discoveries in these research papers will advance personalized medicine and ultimately in clinical settings to improve human health, and quality of life, and create a better world. Cloudam is beyond honored to be part of the significant breakthrough and has powered the research process with HPC solutions.
About Cloudam HPC
Cloudam HPC is a one-stop HPC platform with 300+ pre-installed to deploy immediately. The system can smartly schedule compute nodes and dynamically schedule the software licenses, optimizing workflow and boosting efficiency for engineers and researchers.
Partnered with AWS, Azure, Google Cloud, Oracle Cloud, etc., Cloudam powers your R&D with massive cloud resources without queuing.
You can submit jobs by intuitive templates, SLURM, and Windows/Linux workstations. Whether you are a beginner or a professional, you can always find it handy to run and manage your job.
There is a $30 Free Trial for every new user. Why not register and boost your R&D NOW?