OasysDB is a project that I started in January of this year, and honestly, it has been an incredible learning experience. With it, I've gained quite extensive experience in databases, machine learning algorithms, and low-level programming concepts. But, with this knowledge, I realize that the current design of OasysDB is not enough for production use.
After careful consideration, I've decided to rewrite OasysDB from the ground up. The new version will be designed to incorporate all the essential features needed for a production-ready vector database system.
This includes, but is not limited to:
- Transitioning from an embedded to a client-server model for better scalability and isolation.
- Designing an efficient storage engine tailored for analytical production workloads.
- Implementing concurrent query processing to improve throughput and reduce latency.
- Utilizing advanced vector indexing algorithms for enhanced recall performance, especially in hybrid search scenarios.
- Incorporating an industry-standard query planner and optimizer to enhance query performance.
- Enhancing documentation and testing to ensure the system's robustness and reliability.
Here's a high-level overview of the new architecture:
Progress Update
Today, I started working on the new version of OasysDB. I've established the project structure, implemented the foundational data structures for the collection and storage engine, and set up the initial framework for client-server communication.
I will be posting regular updates (once or twice a week) on my progress, which may include in-depth explorations of the system's technical aspects. If you want to follow along with the development process, you can find the project on GitHub.
An embedded vector database designed to run on edge devices. Lightweight and fast with HNSW indexing algorithm.
📣 Announcement
Many thanks to everyone who supported OasysDB to progress this far. After thorough consideration, I have decided to pivot the project to a new direction. I will be focusing OasysDB to be a more robust vector database solution for production workload with ground-up support for hybrid ANN search algorithms.
This will come with a lot of changes to the project structure, API, and functionality. I will be working on this in the coming weeks and months. I will keep you updated on the progress and the new direction of OasysDB.
~ Edwin
👋 Meet OasysDB
OasysDB is a flexible and easy-to-use vector database written in Rust. It is designed with simplicity in mind to help you focus on building your AI application without worrying about database setup and configuration.
With 3 different runtime modes, OasysDB will accompany you throughout your journey from the early stages of development to…
Conclusion
I'm really excited about the potential of the new OasysDB and the challenges that lie ahead. I believe this overhaul will lead to a robust and scalable vector database system perfect for a wide range of AI applications.
If you're into databases and AI, I encourage you to follow along with the development process as I share my insights, challenges, and victories in this DevLog series. If you have experience in this field, your feedback and suggestions would be greatly appreciated.
Edwin Kys
