Welcome KitOps v0.2! This update brings two major features for working with LLMs, as well numerous smaller enhancements. We are excited to introduce KitOps’ Dev mode - making it possible to test LLMs locally (even if you don’t have an internet connection or GPU).

We also introduce model parts to Kit so you can “chain” ModelKits, simplifying the building of things like adapters for common open source LLMs. These were both features requested by the community 💝

Here's what you need to know about the new additions:

Introducing KitOps Dev Mode

You can try Kit’s new dev mode by using the kit dev command. This initializes and launches a local development portal, allowing you to test various large language models (LLMs). The kit dev command uses the contents of the Kitfile. This initial version of dev mode includes a user-friendly chat and prompt interface and an OpenAI-compatible API, to seamlessly integrate an LLM into your applications.

Getting started is simple:

# Unpack a base LLM
kit unpack ghcr.io/jozu-ai/llama3:8B-text-q4_0 -d ./my-ai-project

# Launch the developer portal
kit dev ./my-ai-project 

After running these commands, you will receive a URL to access the portal through your browser.

Currently dev mode is only available on MacOS, although we plan to expand it to additional platforms and include more inference runtimes and utilities for models, data, and code. File an issue in our GitHub repository telling us what platform we should tackle next, or how to improve the Kit dev command in general - we love community feedback!

More Powerful Model Packaging with Model Parts and Referencing
This release also introduces model parts, a feature to bring even more flexibility to ModelKits. Now you can reference other ModelKits as the base for your ModelKit, or package a more complex AI/ML project into multiple ModelKits, each focused on different models or model parts.

For example, to package a LoRA adapter that you have fine-tuned from the Llama3 base model, your Kitfile would be structured as follows:

model:
  name: my fine-tuned llama3
  path: ghcr.io/jozu-ai/llama3:8B-instruct-q4_0
  parts:
    - path: ./lora-adapter.gguf
      type: lora-adapter

This configuration instructs the kit pack command to package only the LoRA adapter, while the kit unpack and kit pull commands will retrieve your ModelKit and the base model from the referenced ModelKit - so users have everything they need from one command.

Model parts and referencing provide a flexible way to manage and distribute even complex models - for use cases like LoRA adapters, projectors, introducing new parameter sets, and many more. This feature is also helpful for enterprises who want to keep a library of base models, adapters, and even embedding or integration code pre-packaged in ModelKits, for development teams to reference. This can be a great way to provide approved AI/ML packages and “guardrails” for teams as they begin to build with AI/ML.

Get Started Now

We encourage you to explore these new features and take advantage of the other improvements in this release, including bug fixes, documentation enhancements, and performance optimizations. As always, the latest release is available on the KitOps project releases.

Try KitOps v0.2 today and see how these new capabilities can enhance your AI development workflow!

For support or to join the KitOps community, checkout the KitOps Discord server and Star the KitOps GitHub repo to support the project.