We present CLS-Luigi, a framework for synthesizing analytics pipelines that enable prediction and decision-making. Analytics pipelines typically consist of a number of diverse steps ranging from simple preprocessing to informed machine learning and optimization algorithms. Implementing pipelines and selecting algorithms is time-consuming and the performance of selected algorithms is interdependent. CLS-Luigi improves the implementation process by adapting and using a well-established synthesis framework, Combinatory Logic Synthesizer (CLS), to automatically generate pipeline variants based on a repository of typed components that are implemented in Python. Luigi, a pipeline framework developed by Spotify, executes pipeline variants efficiently and allows intermediate results to be shared among similar pipelines to optimize resource utilization. We demonstrate the simplicity, expressive power, and run-time gains of CLS-Luigi through two examples: First, we show that CLS-Luigi has the modeling capabilities to automatically generate machine learning pipelines that cover all algorithms included in AutoSklearn, and delivers high-quality results. Second, we go beyond the capabilities of AutoML by facilitating the consistent synthesis and execution of decision pipelines covering different optimization paradigms. Our framework is easily accessible, open-source, and compatible with most existing tools and Python libraries used in data analytics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CLS-Luigi: Analytics Pipeline Synthesis

  • Anne Meyer,
  • Hadi Kutabi,
  • Jan Bessai,
  • Daniel Scholtyssek

摘要

We present CLS-Luigi, a framework for synthesizing analytics pipelines that enable prediction and decision-making. Analytics pipelines typically consist of a number of diverse steps ranging from simple preprocessing to informed machine learning and optimization algorithms. Implementing pipelines and selecting algorithms is time-consuming and the performance of selected algorithms is interdependent. CLS-Luigi improves the implementation process by adapting and using a well-established synthesis framework, Combinatory Logic Synthesizer (CLS), to automatically generate pipeline variants based on a repository of typed components that are implemented in Python. Luigi, a pipeline framework developed by Spotify, executes pipeline variants efficiently and allows intermediate results to be shared among similar pipelines to optimize resource utilization. We demonstrate the simplicity, expressive power, and run-time gains of CLS-Luigi through two examples: First, we show that CLS-Luigi has the modeling capabilities to automatically generate machine learning pipelines that cover all algorithms included in AutoSklearn, and delivers high-quality results. Second, we go beyond the capabilities of AutoML by facilitating the consistent synthesis and execution of decision pipelines covering different optimization paradigms. Our framework is easily accessible, open-source, and compatible with most existing tools and Python libraries used in data analytics.