Superfaas: a non-invasive FaaS interface for high performance computing systems
摘要
High-throughput computing tasks are a typical class of computational tasks in high-performance computing. They are commonly used for large-scale data analysis in high-energy physics, biomedicine, and other fields. These tasks usually include a large number of small tasks that are independent of each other but have a huge demand for computing resources. In the current HPC resource management pattern, users tend to estimate a certain amount of resource demand and assign tasks to be executed after the resources are satisfied, which often results in a long waiting time for resources. This paper proposes a non-intrusive Function-as-a-Service (FaaS) framework for the supercomputer Internet, called SuperFaaS. SuperFaaS is compatible with existing HPC resource management systems and supports elastic provisioning of HPC computing resources. SuperFaaS ensures the stable execution of tasks through resource reuse, monitoring, and fault-tolerance mechanisms. Tests show that SuperFaaS can achieve the service performance overhead of Openwhisk or even better. Using the drug screening software AutoDock-Vina to calculate 20,000 drug molecule permutations on a real supercomputing system, the results show that SuperFaaS can greatly reduce the total task completion time (including resource waiting time), and the requested resources can achieve more than 95% effective utilization.