Data Pipeline Design for Machine Learning
摘要
The heart of any successful machine learning (ML) project is an efficient and powerful data pipeline. A data pipeline guarantees that data flows seamlessly through all the phases of the ML lifecycle. These phases include data collection and preprocessing, training models, as well as deployment. In this chapter, we cover the conceptual pieces and guidelines for creating data pipelines that are ML-centric. We start with data collection and cleaning. We will have a detailed discussion about various techniques of data cleaning, and we will cover the best practices. Next, we provide an in-depth exploration of feature engineering, starting from fundamental concepts and progressing to more advanced topics. We also address working with text data, which often requires additional preprocessing steps.