🍃Machine Learning Pipeline

In recent years, industries and research groups are developing machine learning pipelines that can automatically handle most of the work, from data collection to model evaluation. This end-to-end pipeline is commonly referred to as "AutoML".

Machine learning pipelines can be categorized in two ways, much like our perfume lines:

• Mini Pipeline - It requires specific data inputs for each run and primarily focuses on model selection, aiming to choose the best model for the given data. Some of these pipelines also incorporate data preprocessing, feature engineering, and hyperparameter optimization. You will find experiments involving TPOT, AutoKeras and MLJar.

• Customized Pipeline - This type of pipeline can be constructed to handle more clients' data and add more complex functionalities.

  • Luigi enables the construction of pipelines that can include data collection, preprocessing, feature engineering, model selection, evaluation, and data drift monitoring, or any other step in your machine learning development process.

  • You will also see simple pipelines built using Airflow and ZenML.