by Caleb Grillo

Fundamentals of BYOC

WarpStream’s Zero Disk Architecture

Typically, cloud data infrastructure products follow one of two deployment models:

1. Fully self-managed, where the customer purchases a software license and support but is ultimately responsible for deploying and managing the software themselves.
2. Fully-hosted SaaS model in which the vendor manages all the infrastructure in their own cloud environment and the customer simply receives an endpoint.

The Bring Your Own Cloud (BYOC) deployment model is a hybrid approach to cloud infrastructure that strikes a balance between these two extremes. Generally, it works like this: the software is split into two different components, a “data plane” (compute + storage) and a “control plane”. The control plane runs in the provider’s environment, and the data plane runs in the customer’s environment.

This deployment model has several benefits:

• Data privacy: Because the data never leaves your environment, you have greater control over who has access to it and under what circumstances.
• Data sovereignty: Data is always stored on resources that you control, so you don’t need to worry about data finding its way to geographical regions where it shouldn’t be.
• Compliance: The data plane is deployed in the customer environment, so strict compliance requirements can be fulfilled, and all traffic can be audited.
• Cost optimization: Because the infrastructure runs in your environment, you can control factors like instance types, networking configurations, and storage classes to optimize costs. You can also take advantage of committed use discounts, reserved instances, and savings plans to further optimize your costs. And perhaps most importantly for a networking-heavy system like Apache Kafka®, the combination of this deployment model and WarpStreams Zero Disk Architecture eliminates virtually all networking fees which can often account for more than 80% of the TCO of a traditional Kafka deployment..
• Control: You have control over the infrastructure that you deploy the software on, so you can choose your own networking topology, instance types, security settings, and storage services that you use.

BYOC makes a lot of sense for mission-critical, data-intensive, and networking-heavy systems like Kafka where throughput is often measured in the hundreds or even thousands of MiBs per second. But historically, BYOC for Kafka has been limited to niche use cases because Kafka (and other equivalent systems) are so stateful and difficult to manage that remotely administering them is almost impossible.

The problem with BYOC for Kafka

Kafka and its derivatives have stateful architectures, with local disks that store partitions that need to be actively managed in order to prevent a variety of issues like: hot partitions, unbalanced storage, under-replicated topic-partitions, etc. This is why there are so many vendors offering a fully-managed Kafka solution, but relatively few that offer a BYOC variant. Managing Kafka in your own environment is difficult enough, but managing Kafka in someone else’s environment is even more challenging.

Since Kafka clusters need to be constantly managed, existing BYOC deployment models for Kafka require providing the vendor with high level access to your environment so their personnel can keep the cluster healthy and mitigate incidents when they inevitably occur. The BYOC vendor often has the ability to manage a huge range of cloud infrastructure, including security policies and resources for your VPC, service accounts, subnetworks, IAM roles, firewall rules, and storage buckets.

But wouldn’t it be better if external access wasn’t required at all?

Zero Access BYOC, secure by default

WarpStream’s primary deployment model is BYOC, but it works a little bit differently from the rest. Unlike most BYOC deployment models, WarpStream was designed to operate with no access to the environment that the Agents and object storage are running in. The only requirement for running the WarpStream Agents is that they have permission to access an object storage bucket in which they can store data, and that they have the ability to establish an outbound connection to the WarpStream Cloud control plane. That’s it. No IAM roles or permissive security policies are required.

This is possible because WarpStream was designed from the ground up with a Zero Disk Architecture with not only full separation of compute and storage, but also separation of data from metadata. This architecture makes managing the WarpStream Agents trivial. The Agents are just stateless compute, so there are no leader elections, no partition rebalances, no disk resizing, and no manual operations required to keep the cluster healthy. WarpStream clusters can be seamlessly scaled in, out, up, or down, with virtually no effort, just like a traditional web server.

WarpStream leverages this Zero Disk Architecture to provide a very high level of service with very little external control by using a shared responsibility model that separates storage, compute, and metadata.

The cloud provider manages the storage, the customer manages the stateless compute (I.E the WarpStream Agents), and WarpStream Cloud manages the metadata / consensus layer. This means that only metadata is transferred from your environment to WarpStream’s, and no raw data ever leaves your environment.

Of course, this zero-access BYOC deployment model does have a tradeoff: WarpStream users are responsible for managing their own (stateless) compute. Fortunately, this is the one thing that everyone running software in the cloud knows how to do: deploy and scale stateless containers! Of course, we do our best to make this easy by providing infrastructure as code primitives like our Terraform Provider and Helm chart.

In exchange for assuming responsibility for deploying and managing the stateless Agents, WarpStream’s users get a deployment model that exposes them to far less risk of a data breach than any other cloud-native model. By design, there is no way for WarpStream Cloud to access your data, even if WarpStream’s cloud account was breached by a hostile actor, or WarpStream was compelled by a government agency.

In fact, WarpStream’s security model is so strong that we even have customers using it for their production workloads in AWS GovCloud regions. While no system can credibly claim to be 100% safe, WarpStream’s design lends itself to a stronger security posture than any of the BYOC products that came before.

Zero trust makes BYOC safe

WarpStream makes the BYOC model safer and more secure than any alternative. This was a deliberate design choice that was made possible by WarpStream’s Zero Disk Architecture. With a zero-trust BYOC model, our customers truly get the best of both worlds: an (almost) fully managed user experience, but with all of the cost and security benefits of running on their own infrastructure.

To learn more about WarpStream’s secure-by-default BYOC deployment model, contact us. Or, if you’re ready to get started, you can sign up and get up and running with WarpStream in just a few minutes. No credit card is required to get started, and your first $400 is on us.