Anmol BaranwalThe time of AI is high especially because of powerful LLMs like GPT-4o and Claude.
Today, I'm covering 21 open source LLM projects that can help you to build something exciting and integrate AI into your project.
As a developer, I can confidently say that AI is not as scary as others make it sound and those who don't learn will get left behind.
Let's cover it all.
By the way, I'm part of Latitude and we're building an open source LLM development platform. You can join the waitlist at ai.latitude.so.
You would be able to do a lot of cool stuff like:
⚡ Deploy prompts as api endpoints.
⚡ Automated evaluations using LLMs.
⚡ Collaborate on prompt engineering.
I'm very confident that you will love it after its release!
1. Vanna - Chat with your SQL database.
Vanna is an MIT-licensed open-source Python RAG (Retrieval-Augmented Generation) framework for SQL generation.
Basically, it's a Python package that uses retrieval augmentation to help you generate accurate SQL queries for your database using LLMs.
It's perfect for developers like me who are not very fond of SQL queries!
Vanna works in two easy steps - train a RAG model on your data, and then ask questions that will return SQL queries that can be set up to run on your database automatically.
You don't need to know how this whole stuff works to use it.
You just have to train a model, which stores some metadata, and then use it to ask questions.
Get started with the following command.
pip install vanna
To make things a little easier, they have built user interfaces that you can use as a starting point for your own custom interface. Find all the interfaces including Jupyter Notebook and Flask.
You can read the docs and you can try this Colab notebook in case you want to see how it works after training.
Watch this demo for a complete walkthrough!
They have 9.5k+ stars on GitHub and are built using Python.
2. Khoj - Your AI second brain.
Khoj is the open source, AI copilot for search. Easily get answers without having to sift through online results or your own notes.
For me, the concept seems exciting and It can help me in researching a lot of projects.
Khoj can understand your Word, PDF, org-mode, markdown, plaintext files, GitHub projects, and even Notion pages.
It's available as a Desktop app, Emacs package, Obsidian plugin, Web app, and Whatsapp AI. Obsidian with Khoj might be the most powerful combo!
You can get started with Khoj locally in a few minutes with the following commands.
$ pip install khoj-assistant
$ khoj
Watch it in action!
Some of the exciting features:
✅ You can share your notes and documents to extend your digital brain.
✅ Your AI agents have access to the internet, allowing you to incorporate real-time information.
✅ You'll get a fast, accurate semantic search on top of your docs.
✅ Your agents can create deeply personal images and understand your speech.
For instance, saying: "Create a picture of my dream house, based on my interests". It will draw this!
Read all the features including shareable chat, online chat, file summarization, and complete details in various categories.
You can read the docs and you can try Khoj Cloud to try it quickly.
Watch the complete walkthrough on YouTube!
It has 12k stars on GitHub and is backed by YCombinator.
3. Flowise - Drag & drop UI to build your customized LLM flow.
Flowise is an open source UI visual tool to build your customized LLM orchestration flow & AI agents.
We shouldn't compare any projects but I can confidently say this might be the most useful one among the projects listed here!
Get started with the following npm command.
npm install -g flowise
npx flowise start
OR
npx flowise start --FLOWISE_USERNAME=user --FLOWISE_PASSWORD=1234
This is how you integrate the API.
import requests
url = "/api/v1/prediction/:id"
def query(payload):
response = requests.post(
url,
json = payload
)
return response.json()
output = query({
question: "hello!"
)}
You can read the docs.
Cloud host is not available so you would have to self-host using these instructions.
Let's explore some of the use cases:
⚡ Let's say you have a website (could be a store, an e-commerce site, or a blog), and you want to scrap all the relative links of that website and have LLM answer any question on your website. You can follow this step-by-step tutorial on how to achieve the same.
⚡ You can also create a custom tool that will be able to call a webhook endpoint and pass in the necessary parameters into the webhook body. Follow this guide which will be using Make.com to create the webhook workflow.
There are a lot of other use cases such as building a SQL QnA or interacting with API. Explore and build cool stuff!
FlowiseAI has 27.5k stars on GitHub and has more than 14k forks so it has a good overall ratio.
4. LLAMA GPT - a self-hosted, offline, ChatGPT like chatbot (Powered by Llama 2).
LlamaGPT is a self-hosted, offline, ChatGPT-like chatbot, powered by Llama 2. It's 100% private and your data doesn't leave your device.
In the recent versions, they have also provided support for Code Llama models and Nvidia GPUs.
You can install it from the Umbrel App Store or you can also install it with Kubernetes.
You can read about the supported models on the docs.
It's very simple to use!
I know, at this point it feels confusing that there are so many ways to run LLM locally. As a developer, I think it's important to evaluate which method works for our situation!
They have 10k+ stars on GitHub and offer 2 packages.
5. LocalAI - free OpenAI alternative.
LocalAI is free, open source, and considered as an alternative to OpenAI.
LocalAI acts as a drop-in replacement REST API that’s compatible with OpenAI (Elevenlabs, Anthropic... ) API specifications for local AI inferencing.
It allows you to run LLMs, generate images, and audio (and not only) locally or on-prem with consumer-grade hardware, supporting multiple model families. The best part is that it does not require GPU.
I never thought that there would be such an option so it's a goldmine for devs who don't want to pay much. Plus, it allows to generate of Text, Audio, Video, and Images and also has voice cloning capabilities. What more do you need?
You can watch the complete walkthrough by Semaphore CI!
There are a lot of integration options and developers have built awesome stuff such as Extension for attaching LocalAI instance to VSCode.
You can read the quickstart guide and how to run it with kubernetes. Find all the resources including how to run it on AWS, k8sgpt, and more.
They have 21k+ stars on GitHub and are on the v2.18 release.
6. Continue - enable you to create an AI software development system.
Continue is one of the best AI code assistants I've seen in my developer journey.
You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains.
You can easily set it up. Below are some of the snapshots while I was installing it.
After you've configured it, you're all set to use all the amazing concepts it provides.
They have a lot of awesome features such as:
Tab to autocomplete code suggestions.
Ask questions about your codebase.
Understand terminal errors immediately.
Kick off actions with slash commands.
Refactor functions where you are coding.
Read about all the features.
You will have to install the VSCode extension from the marketplace and then read the quickstart guide.
You can read the docs.
You can also watch this basic demo on YouTube!
They have 13k+ stars on GitHub and are built using TypeScript.
7. Chat2DB - AI-driven data management platform.
Chat2DB is an AI-first data management, development, and analysis tool.
Its core is AIGC (Artificial Intelligence Generation Code), which can convert natural language into SQL, SQL into natural language, and automatically generate reports, taking efficiency to another level.
Even operations that do not understand SQL can use it to quickly query business data and generate reports.
When you do any operation, it will give you some suggestions.
For instance, when you are doing database development, it will help you generate SQL directly in natural language, give you SQL optimization suggestions, help you analyze SQL performance, analyze SQL execution plan, and can also help you quickly generate SQL test data, system code, etc. It's actually very powerful :)
They have excellent support for multi-data sources and can easily integrate up to 17 different database types including PostgreSQL, MySQL, MongoDB, and Redis.
You can download or try it in the browser.
Let's see some of the exciting features:
✅ Intelligent reports.
✅ Data Exploration.
✅ SQL Development.
You can read the quickstart guide on the official docs.
They have 14k+ stars on GitHub and are on the release v3.2.
8. LibreChat - Enhanced ChatGPT Clone.
LibreChat is a free, open source AI chat platform. This Web UI offers vast customization, supporting numerous AI providers, services, and integrations.
It serves all AI conversations in one place with a familiar interface, and innovative additions, for as many users as you need.
Some of the features are:
✅ Upload and analyze images seamlessly with advanced models like Claude 3, GPT-4, Gemini Vision, Llava, and Assistants.
✅ Chat with files using various powerful endpoints using OpenAI, Azure, Anthropic, and Google.
✅ Multilingual UI with support for 20+ languages.
✅ Diverse Model options including OpenAI, BingAI, Anthropic (Claude), Azure OpenAI, and Google’s premier machine learning offerings.
You can read the quickstart guide to get started.
Watch this video for the complete walkthrough!
They have 15k+ stars on GitHub and offer 4 packages.
9. Lobe Chat - modern-design LLMs/AI chat framework.
An open-source, modern-design ChatGPT/LLMs UI/Framework.
Supports speech-synthesis, multi-modal, and extensible (function call) plugin systems. You can deploy your private OpenAI with one click.
Let's see some of the exciting features of LobeChat:
✅ Multi-Model Service Provider Support.
They have expanded our support to multiple model service providers, rather than being limited to a single one.
Find the complete list of 10+ model service providers that they support.
✅ Assistant Market.
In LobeChat's Assistant Market, creators can discover an innovative community that brings together numerous carefully designed assistants.
There are so many awesome applications there. WOW!
✅ Model Vision Recognition.
LobeChat now supports large language models with visual recognition capabilities such as OpenAI's gpt-4-vision, Google Gemini Pro vision, and Zhipu GLM-4 Vision, enabling LobeChat to have multimodal interaction capabilities.
Users can easily upload or drag and drop images into the chat box, and the assistant will be able to recognize the content of the images and engage in intelligent conversations.
✅ Text to Image Generation.
You can directly utilize the Text-to-image tool during conversations with the assistant.
By using the power of AI tools such as DALL-E 3, MidJourney, and Pollinations, assistants can now implement it properly.
✅ Local Large Language Model (LLM) Support.
With the powerful infrastructure of Ollama AI and the community's collaborative efforts, you can now engage in conversations with a local LLM (Large Language Model) in LobeChat!
By running the following Docker command, you can experience conversations with a local LLM in LobeChat.
docker run -d -p 3210:3210 -e OLLAMA_PROXY_URL=http://host.docker.internal:11434/v1 lobehub/lobe-chat
✅ Progressive Web App (PWA).
They have adopted Progressive Web App PWA technology, which is a modern web technology that elevates web applications to a near-native app experience.
✅ Custom Themes.
Read about all of the features and use cases.
You can self-host or deploy it using docker. The ecosystem of lobe chat provides 4 packages: lobehub/ui, lobehub/icons, lobehub/tts, and lobehub/lint.
They also provide plugins market where you can find lots of useful plugins that can be used to introduce new function calls and even new ways to render message results. If you want to develop your own plugin, refer to 📘 Plugin Development Guide in the wiki.
You can read the docs.
You can check the live demo. It's pretty cool!
They have 35k+ stars on GitHub with more than 500 releases.
10. MindsDB - The platform for customizing AI from enterprise data.
MindsDB is the platform for customizing AI from enterprise data.
With MindsDB, you can deploy, serve, and fine-tune models in real-time, utilizing data from databases, vector stores, or applications, to build AI-powered apps - using universal tools developers already know.
With MindsDB and its nearly 200 integrations to data sources and AI/ML frameworks, any developer can use their enterprise data to customize AI for their purpose, faster and more securely.
You can read the docs and quickstart guide to get started.
They currently support a total of 3 SDKs that is using using Mongo-QL, Python, and JavaScript.
There are several applications of MindsDB such as integrating with numerous data sources and AI frameworks so you can easily bring data and AI together to create & automate custom workflows.
The other common use cases include fine-tuning models, chatbots, alert systems, content generation, natural language processing, classification, regressions, and forecasting. Read more about the use cases and each of them has an architecture diagram with a little info.
For instance, the chatbot architecture diagram with MindsDB. You can read about all the solutions provided along with their SQL Query examples.
// SQL Query Example for Chatbot
CREATE CHATBOT slack_bot USING database='slack',agent='customer_support';
Just to tell you about the overall possibilities, you can check out How to Forecast Air Temperatures with AI + IoT Sensor Data. Exciting right :)
They have 25.4k+ stars on GitHub and are on the v24.7.2.0 with more than 200 releases. By the way, this is the first time I've seen 4 parts in any release as I always followed the semantic release.
11. AutoGPT - more exciting than ChatGPT.
At the core of AutoGPT lies its primary project, a semi-autonomous agent driven by large language models (LLMs), designed to perform any tasks for you.
The AutoGPT project consists of four main components:
- The Agent – also known as just "AutoGPT"
- The Benchmark – AKA agbenchmark
- The Forge
- The Frontend
Read on how you can set up AutoGPT using the OpenAI key.
You can see this YouTube video by Fireship on what is AutoGPT.
You can also watch this AutoGPT tutorial by Sentral Media.
You can read the docs and check out the project board on what things are under development right now.
Even if you don't know much about AI, you can try AutoGPT to understand how you can save time and build cool stuff.
They have 164k+ stars on GitHub due to such an excellent use case and automation capabilities.
12. reor - self organizing AI note-taking app.
One of the most exciting projects that I've seen so far, especially because it runs models locally.
Reor is an AI-powered desktop note-taking app. It automatically links related notes, answers questions on your notes, and provides semantic search.
Everything is stored locally and you can edit your notes with an Obsidian-like markdown editor. The project hypothesizes that AI tools for thought should run models locally by default.
Reor stands on the shoulders of the giants Ollama, Transformers.js & LanceDB to enable both LLMs and embedding models to run locally. Connecting to OpenAI or OpenAI-compatible APIs like Oobabooga is also supported.
I know you're wondering How can it possibly be
self-organizing?
a. Every note you write is chunked and embedded into an internal vector database.
b. Related notes are connected automatically via vector similarity.
c. LLM-powered Q&A does RAG on the corpus of notes.
d. Everything can be searched semantically.
You can watch the demo here!
One way to think about Reor is as a RAG app with two generators: the LLM and the human. In Q&A mode, the LLM is fed retrieved context from the corpus to help answer a query.
Similarly, in editor mode, we can toggle the sidebar to reveal related notes retrieved from the corpus. This is quite a powerful way of augmenting your thoughts by cross-referencing ideas in a current note against related ideas from your digital collection.
You can read the docs and download from the website. Mac, Linux & Windows are all supported.
They have also provided starter guides to help you get started.
You can also watch this walkthrough!
They have 6.5k stars on GitHub and are built using TypeScript.
13. Leon - your open source personal assistant.
Leon is an open source personal assistant who can live on your server. You're definitely curious, right? 😅
He does stuff when you ask him to. You can even talk to him and he will revert by talking to you. Similarly, you can also text him!
If you are a developer (or not), you may want to build many things that could help in your daily life. Instead of building a dedicated project for each of those ideas, Leon can help you with his Skills structure.
If you want to, Leon can communicate with you by being offline to protect your privacy. This is a list of skills that Leon can do for now.
You should read the story behind Leon. You can also watch this demo to learn more about Leon.
This is the High-level architecture schema of Leon.
You can install Leon by using this command.
# install leon global cli
npm install --global @leon-ai/cli
# install leon
leon create birth
You can read the docs.
Appwrite is one of the sponsors which says a lot about its overall impact.
It has 15k stars on GitHub and has released some drastic changes recently so make sure to read the docs with extra caution.
14. Instrukt - Integrated AI in the terminal.
Instrukt is a terminal-based AI-integrated environment. It offers a platform where users can:
- Create and instruct modular AI agents.
- Generate document indexes for question-answering.
- Create and attach tools to any agent.
Instruct them in natural language and, for safety, run them inside secure containers (currently implemented with Docker) to perform tasks in their dedicated, sandboxed space.
It's built using Langchain, Textual, and Chroma.
Get started with the following command.
pip install instrukt[all]
There are a lot of exciting features such as:
✅ A terminal-based interface for power keyboard users to instruct AI agents without ever leaving the keyboard.
✅ Index your data and let agents retrieve it for question-answering. You can create and organize your indexes with an easy UI.
✅ Index creation will auto-detect programming languages and optimize the splitting/chunking strategy accordingly.
✅ Run agents inside secure docker containers for safety and privacy.
✅ Integrated REPL-Prompt for quick interaction with agents, and a fast feedback loop for development and testing.
✅ You can automate repetitive tasks with custom commands. It also has a built-in prompt/chat history.
You can read about all the features.
You can read the installation guide.
You can also debug and introspect agents using an in-built IPython console which is a neat little feature.
Instrukt is licensed with an AGPL license meaning that it can be used by anyone for whatever purpose.
It is safe to say that Instrukt is an AI commander for your terminal.
It is a new project so they have around 240 stars on GitHub but the use case is damn good.
15. Quivr - RAG Framework for building GenAI Second Brains.
Quivr, your second brain, utilizes the power of GenerativeAI to be your personal assistant! You can think of it as Obsidian but turbocharged with AI powers.
Quivr is a platform that enables the creation of AI assistants, referred to as Brain. These assistants are designed with specialized capabilities like some can connect to specific data sources, allowing users to interact directly with the data.
While others serve as specialized tools for particular use cases, powered by Rag technology. These tools process specific inputs to generate practical outputs, such as summaries, translations, and more.
Watch a quick demo of Quivr!
Some of the amazing features are:
✅ You can choose the type of Brain you want to use, based on the data source you wish to interact with.
✅ They also provide a powerful feature to share your brain with others. This can be done by sharing with individuals via their emails and assigning them specific rights.
✅ Quivr works offline, so you can access your data anytime, anywhere.
✅ You can access and continue your past conversations with your brains.
✅ But the best one that I loved is that you can literally install a Slack bot. Refer to this demo to see what you can do. Very cool!
Anyway, read about all the awesome stuff that you can do with Quivr.
You can read the installation guide and 60 seconds installation video. I really loved this idea!
You can read the docs.
They have also provided guides on how to deploy Quivr with Vercel, Porter, AWS, and Digital Ocean.
They could provide a better free tier plan but it's more than enough to test things on your end.
It has 30k+ Stars on GitHub with 220+ releases which means they're constantly improving.
16. Open Interpreter - natural language interface for terminal.
Open Interpreter lets LLMs run code (Python, Javascript, Shell, and more) locally. You can chat with Open Interpreter through a ChatGPT-like interface in your terminal by running $ interpreter after installing.
This provides a natural-language interface to your computer's general-purpose capabilities:
✅ Create and edit photos, videos, PDFs, etc.
✅ Control a Chrome browser to perform research Plot, clean, and analyze large datasets.
I don't know about you, but their website made me say WOW!
Quickstart using this command.
pip install open-interpreter
// After installation, simply run:
interpreter
You can read the quickstart guide.
You should read about the comparison to ChatGPT's Code Interpreter and the commands that you can use.
You can read the docs.
Open Interpreter works with both hosted and local language models. Hosted models are faster and more capable, but require payment while local models are private and free but are often less capable. Choose based on your use case!
They have 48k+ stars on GitHub and are used by 300+ developers.
17. CopilotKit - 10x easier to build AI Copilots.
You will agree that it's tough to add AI features in React, that's where Copilot helps you as a framework for building custom AI Copilots.
You can build in-app AI chatbots, and in-app AI Agents with simple components provided by Copilotkit which is at least 10x easier compared to building it from scratch.
You shouldn't reinvent the wheel if there is already a very simple and fast solution!
They also provide built-in (fully-customizable) Copilot-native UX components like <CopilotKit />, <CopilotPopup />, <CopilotSidebar />, <CopilotTextarea />.
Get started with the following npm command.
npm i @copilotkit/react-core @copilotkit/react-ui
This is how you can integrate a Chatbot.
A CopilotKit must wrap all components which interact with CopilotKit. It’s recommended you also get started with CopilotSidebar (you can swap to a different UI provider later).
"use client";
import { CopilotKit } from "@copilotkit/react-core";
import { CopilotSidebar } from "@copilotkit/react-ui";
import "@copilotkit/react-ui/styles.css";
export default function RootLayout({children}) {
return (
<CopilotKit url="/path_to_copilotkit_endpoint/see_below">
<CopilotSidebar>
{children}
</CopilotSidebar>
</CopilotKit>
);
}
You can read the docs and check the demo video.
You can integrate Vercel AI SDK, OpenAI APIs, Langchain, and other LLM providers with ease. You can follow this guide to integrate a chatbot into your application.
The basic idea is to build AI Chatbots very fast without a lot of struggle, especially with LLM-based apps.
You can watch the complete walkthrough!
CopilotKit has recently crossed 7k+ stars on GitHub with 300+ releases.
18. GPT Engineer - AI builds what you ask.
GPT-engineer lets you specify software in natural language, sit back, and watch as an AI writes and executes the code, and you can ask the AI to implement improvements.
It's safe to say that it's an engineer who doesn't need a degree 😅
It's a commercial project for the automatic generation of web apps. It features a UI for non-technical users connected to a git-controlled codebase.
I know this feels confusing, so watch the below demo to understand how you can use GPT Engineer.
You can get started by installing the stable release using this command.
python -m pip install gpt-engineer
By default, gpt-engineer expects text input via a prompt file. It can also accept image inputs for vision-capable models. This can be useful for adding UX or architecture diagrams as additional context for GPT engineer. Read about all the awesome features.
If you want a complete walkthrough, watch this awesome demo by David!
I recommend checking out the roadmap to understand the overall vision.
They have 51k+ stars on GitHub and are on the v0.3 release.
19. Dalai - the simplest way to run LLaMA and Alpaca locally.
Dalai lets you run LLaMA and Alpaca on your computerP and is powered by llama.cpp, llama-dl CDN, and alpaca.cpp.
Dalai runs on operating systems such as Linux, Mac, and Windows so that's a plus point!
Dalai is also an NPM package:
- programmatically install
- locally make requests to the model
- run a dalai server (powered by socket.io)
- programmatically make requests to a remote dalai server (via socket.io)
You can install the package using the below npm command.
npm install dalai
You can read the memory requirements and how to archive elsewhere rather than home directory.
They have 13k stars on GitHub and are still in the very early stages.
20. OpenLLM - run LLMs as OpenAI compatible API endpoint in the cloud
OpenLLM lets developers run any open-source LLMs as OpenAI-compatible API endpoints with a single command.
⚡ Build for fast and production usage.
⚡ Support llama3, qwen2, gemma, etc, and many quantized versions full list.
⚡ OpenAI-compatible API & includes ChatGPT like UI.
⚡ Accelerated LLM decoding with state-of-the-art inference backends.
⚡ Ready for enterprise-grade cloud deployment (Kubernetes, Docker, and BentoCloud).
Get started with the following command.
pip install openllm # or pip3 install openllm
openllm hello
OpenLLM provides a chat user interface (UI) at the /chat endpoint for an LLM server. You can visit the chat UI at http://localhost:3000/chat and start different conversations with the model.
OpenLLM supports LLM cloud deployment via BentoML, the unified model serving framework, and BentoCloud, an AI inference platform for enterprise AI teams.
If you don't know, BentoCloud provides a fully managed infrastructure optimized for LLM inference with autoscaling, model orchestration, observability, and many more, allowing you to run any AI model in the cloud.
You can read about the supported models and how to start the LLM server.
Explore docs as you can also chat with a model in the CLI using openllm run and specifying model version - openllm run llama3:8b.
For people like me who love exploring walkthroughs, watch this demo by Matthew!
They have 9k+ stars on GitHub and have 100+ releases so it's growing at a rapid pace.
21. Unsloth - Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory.
Unsloth makes finetuning large language models like Llama-3, Mistral, Phi-3, and Gemma 2x faster, use 70% less memory, and with no degradation in accuracy!
✅ What is finetuning?
If we want a language model to learn a new skill, a new language, some new programming language, or simply want the language model to learn how to follow and answer instructions like how ChatGPT functions, we do finetuning!
Finetuning is the process of updating the actual brains of the language model through some process called back-propagation. But, finetuning can get very slow and very resource intensive.
Unsloth can be installed locally or through another GPU service like Google Colab. Most use Unsloth through the interface Google Colab which provides a free GPU to train with.
Some of the things that stand out:
✅ Open source trains 5x faster, and the pro version claims to be 30x faster.
✅ No approximation methods are used resulting in a 0% loss in accuracy.
✅ No change of hardware, Works on Linux and Windows via WSL.
You can read the installation instructions and performance benchmarking tables on the website.
You can read the docs and all the uploaded models on Hugging Face directly.
They have also provided a detailed guide on How to Finetune Llama-3 and Export to Ollama.
They have 12.5k+ stars on GitHub and it's an efficient solution.
I hope you learned something new!
I believe that learning to use these powerful LLM is a choice and it's on you (as a developer) to find better productivity solutions for your use case.
Have a great day! Till next time.
| If you like this kind of stuff, please follow me for more :) |
  
|
|---|
Follow Latitude for more content like this.
