Composable Parallelism in Data Processing Tasks
摘要
Data analytics is an inevitable component for any major organization that works with data, including banks, service aggregators, governments, insurance providers, and advertising companies. There are two primary approaches to parallelizing a data analytics engine: by requests (serial execution within a request) or by data (serializing and distributing requests). Running too many requests simultaneously can exceed memory bandwidth limits, while dedicating an entire machine to a single request may be inefficient due to data size and underutilization of processor cores. Can we combine these approaches to overcome their limitations—processing multiple requests concurrently while enabling data parallelism within each request? We evaluate composable threading solutions using the New York City Taxi benchmark to address this question.