In today’s world, where almost every company is embracing artificial intelligence (AI) and machine learning (ML) into their software offering, maintaining two separate pipelines for ML-powered software systems and conventional software projects can pose challenges. This also introduces friction within the team, which can slow down the development and deployment process.

In this article, we will discuss the cause behind adopting separate pipelines for conventional software projects and ML-powered projects, the challenges introduced by such construct, and how developers can turn their DevOps pipeline into an MLOps pipeline and become more efficient by embracing ModelKits.

Why can’t we use the same pipeline for ML products and conventional software projects?

There are three main reasons that necessitate separate pipelines for deploying ML products and conventional software engineering projects:

• Nature of projects
• Individual expertise
• Size and complexity

Nature of projects
ML projects are fundamentally different from general software projects in the sense that training machine learning models are an iterative process, and the model artifacts are closely tied to the data and code used to produce them.

In contrast, DevOps principles assume that a software application consists of code (the dynamic component) and static files (images, videos, etc.).

Due to these differences in the fundamental principles of traditional software development and data science projects, DevOps and MLOps are usually treated as two different concepts and teams.

Individual expertise
Most ML tools in the ecosystem are built with the premise that they will be used by individuals (data scientists, data engineers, machine learning engineers, etc.) with a fundamental understanding of the machine learning lifecycle, machine learning models, its different components, and how they interact with one another. As a result, even though they are technically proficient and possess a solid understanding of software and infrastructure, the DevOps engineering team is not equipped to deploy ML projects into production.

Size and complexity
Artifacts like trained models and data tend to bloat ML project repositories. Furthermore, the close relationship between data, code, and model makes tracking the code and data used to produce a particular model version complex and vice versa. While conventional DevOps tools like Git are great for tracking code changes, they fail to accommodate the large-sized artifacts in ML projects and cannot be easily adapted to deploy machine learning models.

Shortcomings of the MLOps pipeline

The above challenges have led organizations to adopt a separate MLOps pipeline. However, implementing a separate MLOps pipeline has some unique challenges of its own:

• Data management and standardization issues
• Security and compliance issues
• Complex model lifecycle management

Data management and standardization issues
The data used to build ML models changes or updates frequently. Maintaining quality and consistency across different stages of the pipeline is challenging, often leading to a pipeline that utilizes numerous tools.

The MLOps ecosystem is developing; new tools are frequently introduced, and it is rapidly evolving. Some tools package the ML product using their proprietary standards, making it difficult to use them with other open source tools that handle other aspects of the pipeline not supported by the tool and locking the customers into a closed ecosystem.

All of these issues lead to integration challenges between the development and deployment team and the pipeline.

Security and compliance issues
Managing user data in accordance with regulations throughout different stages of the pipeline can be challenging. Proprietary tools can worsen this problem, as they might not meet security standards for user data and are difficult to audit.

Complex model lifecycle management
ML models are unique in that, after deployment, the team must monitor not only the infrastructure (network, scalability, etc.) and software components (code quality, bugs, vulnerabilities, etc.) but also the model itself for drift. Addressing this challenge often requires additional tools, which can further complicate the pipeline and amplify the issues mentioned earlier.

These peculiarities make it difficult for ML stakeholders (data scientists, data engineers, etc.) to efficiently collaborate with the deployment team within the organization.

Side-effects of maintaining two separate pipelines

Peculiarities of ML software have forced companies to keep two teams and maintain separate pipelines to host their traditional software engineering projects and machine learning projects. This has resulted in:

• Increased cost
• Coordination challenges
• Increased technical debt

Increased cost
Hiring and maintaining payroll for two different teams will increase the cost for a team/company.
Furthermore, the two teams will need separate resources (servers, network bandwidth, credentials for paid softwares, etc.), which increases the cost.

Coordination challenges
When there are two different operations teams within an organization, knowledge can become siloed, and teams may not effectively resolve issues spanning both domains due to communication barriers.

Bottlenecks in communication can also introduce delays in deployment, resulting in loss of profits or customers due to unsatisfactory customer experience.

Increased technical debt
ML team often consists of data scientists who are domain experts in their fields and can easily run experiments for model training and model development, build visualization for model monitoring, and produce a stellar model but may lack knowledge of software testing, automated testing, design patterns, continuous deployment, continuous delivery, and good coding practices. This may introduce inefficiencies that may hurt team performance in the long run.

As discussed previously, DevOps tools are great at building and maintaining conventional software engineering projects, but they were not built with machine learning workflow in mind. In fact, the term DevOps was coined in 2009 when managing machine learning models was not the center of attention. They are not ideal for managing models, maintaining complex data workflows, data versioning, model packaging, and model deployment. Consequently, adopting DevOps tools to ML projects necessitates creating workarounds that may also include manual steps and non-optimal code, which is not ideal.

A bird' s-eye view of the MLOps pipeline reveals that if you can separate the experimental part—the part where you experiment with different models and data to come up with a model that is good enough to deploy—and then tie data, code, and model together, the rest of the pipeline is very similar to the DevOps pipeline.

How can you achieve this coherency and seamless integration between data, model, and code? ModelKits can help you tie your data, model, and code together and then use them with your preferred tool or cloud service for inference.

Let’s look at the MLOps workflow with ModelKits to learn more about the process.

MLOps workflow with ModelKits

With Modelkits, there are three distinct steps in your MLOps pipeline:

• Training and experimentation
• Package, validate, and deployment
• Inference

Let’s look at each of these steps.

Training and experimentation
ModelKits allow you to use your favorite tools for experimentation, model training, and experiment tracking. This means you can continue using your current toolkit, Jupyter Notebook, version control software, model evaluation library, or anything else, including your current feature stores and feature engineering pipeline, without any modifications. This makes it easier for data scientists and machine learning engineers in the team to adopt ModelKits.

Packaging, validation, and deployment
Once you train the model, you will need to package the model artifact, code, and associated data. You can do so in four steps:

1. Write a Kitfile, which is very similar to writing any configuration file.
2. Pack the content of the Kitfile in a ModelKit using the kit pack available in the Kit CLI.
3. Push the ModelKit to a remote container registry (could be GitHub Registry or Jozu Hub: A ModelKit-first registry.

These steps above will package and version your code, data, and model. For added security, all artifacts included in ModelKit are signed. Using ModelKit doesn’t require any prior machine learning knowledge, making it ideal for DevOps engineers. Furthermore, ModelKit uses open source standards and is easily compatible with open source tools commonly used by DevOps and MLOps engineers, making it easy to adopt.

Inference
Finally, you can use your existing inference service or platform to deploy machine learning models. All you need to do is unpack the necessary component of the ML model from the ModelKit, that is, the model, and either store it in a location (it could be a directory in a cloud platform or your local server) where the application code can find it or point the application code to the model location. Your customers can now take advantage of the deployed machine learning models.

If you carefully observe the above steps, you will find that they are similar to the process involved in deploying conventional software. You experiment and build locally, commit and push your code changes using version control software, and then deploy it using a continuous integration tool.

You can replicate the above steps to deploy your software engineering project as well. This is the beauty of KitOps. Adopting KitOps allows you to use the same pipelines for normal software engineering projects and machine learning projects.

Step by Step implementation

Let’s look at a simple DevOps pipeline for deploying a simple Python app or script.

name: Python DevOps Pipeline
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.x
      - name: Install Dependencies
        run: pip install -r requirements.txt

      - name: Run Tests
        run: pytest tests/

      - name: Deploy App
        run: python src/deploy.py

The above file runs as soon as you push your code to GitHub and does the following:

1. Get the Ubuntu OS (# ubuntu-latest).
2. Get the code from the repository (# Checkout repository).
3. Install Python (# Set up Python).
4. Install required dependencies from the requirements file (# Install dependencies).
5. Run tests (# Run tests).
6. Run the deployment script (# Deploy app).

The pipeline works well for deploying non-ML applications, but Git fails to track ML-based applications that depend on multiple datasets and model artifacts. Hence, the pipeline fails for ML-based applications. Let’s modify this existing DevOps pipeline a bit to accommodate both ML and non-ML applications. We will be using ModelKit and KitOps.

With KitOps and ModelKits, you will first make the local changes and then write the Kitfile (you will need to install Kit and login to an OCI registry):

manifestVersion: v1.0.0
package:
  name: Python Example
  description: 'A simple python app'
  license: Apache-2.0
  authors: [Jozu]
code:
  - path: .
    description: Sample app code

It is important to note that the Kitfile above tracks only the code. Nevertheless, you can also register your data and model using the same Kitfile. You will find a sample Kitfile tracking data and model below:

manifestVersion: v1.0.0
package:
  name: Python Example
  description: 'A simple python app'
  license: Apache-2.0
  authors: [Jozu]
code:
  - path: .
    description: Sample app code
model:
  name: FilmSLM
  description: Film photography Q&A model using Mistral-7B
  framework: Mistral-7B
  license: Apache-2.0
  path: ./models/film_slm:champion
  version: 1.2.6
datasets:
  - description: Forum postings from sites like rangefinderforum, PhotographyTalk, and r/AnalogCommunity
    name: training data
    path: ./data/forum-to-2023-train.csv
  - description: validation data
    name: validation data
    path: ./data/test.csv

After writing the Kitfile, you can package the contents into a ModelKit using:

kit pack . -t YOUR_CONTAINER_REGISTRY_URL/APP_NAME:TAG

You can push the ModelKit to your registry using kit push:

kit push YOUR_CONTAINER_REGISTRY_URL/APP_NAME:TAG

Your app is now ready for testing. On your test server, you can unpack the ModelKit using:

kit unpack YOUR_CONTAINER_REGISTRY_URL/APP_NAME:TAG

If you want to unpack code, data, or model selectively, specify the relevant flag: --code, --model, and --datasets. For Example,

kit unpack YOUR_CONTAINER_REGISTRY_URL/APP_NAME:TAG --code

You can now run the original pytest command to test your code:
pytest

Once the tests pass, you can build a container image, package it along with the code, and push it to the remote registry.

kit pack . -t YOUR_CONTAINER_REGISTRY_URL/APP_NAME:TAG
kit push YOUR_CONTAINER_REGISTRY_URL/APP_NAME:TAG

Or, you could just write a Dockerfile and package it along with your code.

Finally, you can unpack the code from your remote registry to your deployment server and deploy it.

kit unpack YOUR_CONTAINER_REGISTRY_URL/APP_NAME:TAG --code
docker run <CONTAINER_NAME>:<TAG>

Maintaining two separate pipelines for essentially the same functionality can introduce communication overhead, technical debt, and waste company resources. As a result, it is wiser to bind the MLOps and DevOps pipelines into a single unit for efficient deployment of software engineering and machine learning projects. This can be easily achieved by embracing KitOps and ModelKit.

Furthermore, compatibility with other open source tools and enterprise support from Jozu make it easier to adopt ModelKit. The fastest way to reach them for more information about their projects and cloud offerings is through their official Discord channel.