Jesse WilliamsContrary to traditional software projects, building an ML project is an iterative process and involves numerous steps like identifying your business goals, processing data (data collection, data preprocessing, feature engineering), developing your model (training, tuning, evaluation), deploying your model (inference, prediction), and monitoring your model.
Each step in the machine learning workflow introduces additional complexity and new tools to handle those complexities, which can quickly lead to a convoluted dev stack. Due to this, it is easy to lose track of the end goal: creating and deploying a model to production.It’s estimated that 87% of machine learning models never make it to production. To bridge this gap, MLOps has been introduced into enterprise AI development processes.
MLOps helps increase team efficiency and efficacy when working on ML projects. There are multiple vendors building MLOps solutions, but in this blog we are going to explore 10 open source MLOps tools that can help build an effective (and flexible) MLOps pipeline.
10 open source tools to build an MLOps pipeline
This article will focus on the following open source tools, their key features, and their pros and cons:
- KitOps
- Hydra
- Data Version Control (DVC)
- Airflow
- Continuous Machine Learning (CML)
- Hyperopt
- Weights and Biases
- MLFlow
- NannyML
- MetaFlow
- KitOps
Most MLOps libraries and frameworks handle data, code, ML models, and artifacts in isolation. In addition to this, closed source MLOps tools use vendor specific packaging standards that aren’t compatible with other tools and services. Because of this, teams are required to use a unique tool for handling each component, or repackage the project at each handoff, wasting a ton of time and making AI project development handoffs difficult. This introduces complexity since a different tool needs to be installed to handle the components of an ML project.
KitOps, however, treats all components in an ML project as a single software unit, making it easier to package, version, and track those components. This is called a ModelKit, and is based on the OCI-standard (similar to Docker and Kubernetes), which makes ModelKits compatible with almost every software development tool. With KitOps, it is possible to unpack individual components (data, model, or code) and work on them. This makes it easier to collaborate on the project and simplifies the dependency management. KitOps supports a majority of the tools in the ML ecosystem and has beginner-friendly documentation, making it easier to switch.
⭐️ Star KitOps so you can find it again later ⭐️
- Hydra
Hydra is a configuration management tool developed by Meta. It allows users to specify configurations via a file or the command line and supports hierarchical configurations. Hydra is extremely lightweight and easy to learn, making it ideal for beginners and experts. Furthermore, it allows users to run multiple jobs with a single command, which is ideal for the hyperparameter tuning of deep learning models.
However, depending on the use case, having a separate tool for configuration management can result in complex ML workflows.
- Data Version Control *****(DVC)*
As Git helps you with code versions and the ability to roll back to previous versions for code repositories, DVC has built-in support for tracking your data and model. This helps machine learning teams reproduce the experiments run by your fellows and facilitates collaboration. DVC is based on the principles of Git and is easy to learn since the commands are similar to those of Git. Other benefits of using DVC include:
- It supports cloud and on-premise hardware storage services, making it ideal for small and large teams.
- The availability of a VS Code extension makes it easy to use.
However, for large datasets, storing multiple versions of the data can lead to storage and performance overhead.
- Airflow
An integral part of an ML project is data acquisition and data transformation into the required format. This involves creating ETL (extract, transform, load) pipelines and running them periodically. Airflow is an open source platform that helps engineers create and manage complex data pipelines. Furthermore, the support for Python programming language makes it easy for ML teams to adopt Airflow.
However, despite the benefits and support for Python, learning to use Airflow has a steep learning curve, making it less beginner-friendly.
- Continuous Machine Learning *****(CML)*
Continuous Machine Learning (CML) is another great tool for ML teams by iterative (creator of DVC) that implements continuous integration and delivery (CI/CD) with a focus on ML. It automates model training, model evaluation, and comparison of ML models built as a result of ML experiments.
Implementing CML is similar to writing GitHub Actions, hence it is beginner-friendly. Additionally, the output of steps in the CML is displayed as GitHub comments, which makes it easier to comment and collaborate.
- Hyperopt
ML models involve numerous hyperparameters whose values can influence the model's overall performance. The only way of finding the best value for those parameters is to run the models with different sets of hyperparameter values, record the performance, and compare them. When done manually, this can be cumbersome. Hyperopt helps in automated hyperparameter optimization.
An advantage of hyperopt is that it can be run on a single computer or in a distributed setting.
- Weights and Biases
Weights and Biases (W&B) ****is a tool for visualizing and tracking machine learning experiments. It supports major machine learning frameworks such as TensorFlow and PyTorch. Its key features include:
- Dataset and model parameters visualization
- Model tracking and data tracking
- Hyperparameter optimization for deep learning models
- Create reports using the artifacts tracked using weights and biases
It is a combination of DVC, CML, and Hyperopt. But, unlike those tools, Weights and Biases is only free for experimentation (single user) and academics. If you want to use it for a commercial product, you can host it on your own premises, if you have the compute resources, or use W&B’s cloud infrastructure.
- MLflow
MLflow is an open source MLOps tool that allows users to manage the entire life cycle of machine learning models. It has four key components:
- MLflow tracking: Tracks parameters, code, and results in machine learning experiments with support for hyperparameter tuning
- MLflow projects: Packaging format for ML code for reproducible runs
- MLflow model registry: A centralized registry for storing models
- MLflow models: A packaging format for quickly deploying ML models to cloud platforms such as Azure ML and AWS SageMaker
Since it supports all major ML libraries, it can be an excellent tool for your organization. However, engineers will need to spend significant time learning about the tool and its key features before fully utilizing it.
- NannyML
In machine learning projects, it is difficult to estimate whether improvement in model metrics will result in a positive change in business value. Furthermore, a model’s performance can change with time. NannyML is an open source Python library that focuses on production monitoring and allows users to detect drifts (data and label), check data quality, estimate post-deployment model performance, and intelligently generate an alert if the drift is likely to impact model performance.
NannyML has a free and premium version. The core features of the monitoring tools are available on both versions, but there are some features that are unique to the premium version. A detailed comparison between both versions is available for your review.
- Metaflow
Metaflow is an open source Python library that allows engineers to build and manage ML projects. It focuses on rapid prototyping and reducing time from development to production. It makes the job of ML data scientists easier by taking care of the low-level infrastructure: data, compute, orchestration, and versioning.
How do you choose the right tools to build an MLOps pipeline?
Now that we’ve covered some of your options, how do you select the best MLOps tools for your use case? Well, there are a few criteria you could use:
- Team expertise
- Budget
- Scope
- Documentation and support
Let’s go through each of them in detail.
Team expertise
The expertise of your current ML team hugely impacts the choice of tools you can use. For instance, if your team has expertise in Python but the tool doesn’t support Python language, you should look for an alternative. Keep in mind that the tools listed above are open source and managing an open source tool can have a steep learning curve.
Budget
Some tools like MLflow and Weights and Biases have freemium and premium versions. Before making a decision, it is better to check if the freemium version has all the features you need. If it doesn’t and you still want to use the tool, consider your budget before deciding. The tools we’ve listed above are all open source and can be used for free, however, there are limitations to many of the open source projects that should be explored before making the decision to adopt the tool.
Scope
Tools like DVC and Hyperopt are very specific in what they can do. DVC is only used for tracking data and models, while hyperopt is only used for optimizing hyperparameters—and there are other tools that try to do everything. If your requirements are fulfilled by lightweight tools that provide specific functionalities, it is better to stick with them as they will result in less storage and performance overhead.
Documentation and support
Regarding software products, choosing a tool with good documentation and support with fewer features is better than a tool with poor documentation and support and more features. With good documentation, you can always extend the tool with missing features, but if the documentation is poor, you will likely get stuck if a problem arises.
Before choosing an open source tool, we encourage you to read the documentation, visit the respective Slack or Discord channel, and ask questions to gauge the response time and quality of support delivered by the team behind the tool. The list of tools we’ve given in this post are well documented and have active communities and contributors.
