Stream processing made even more operationally mundane

We previously launched embedded support for Benthos in the WarpStream Agent, which enables WarpStream users to use Benthos without managing any additional infrastructure.

Sign up for a free WarpStream trial.

Today, we’re excited to announce that we have taken this feature a step further with WarpStream Managed Data Pipelines. A new feature for WarpStream BYOC clusters, Managed Data Pipelines provide a fully-managed SaaS user experience for Benthos, without sacrificing any of the cost benefits, data sovereignty, or deployment flexibility of the BYOC deployment model.

Deploy multiple Pipelines from the WarpStream console

Benthos is a lightweight stream processing framework that offers much of the functionality of Kafka Connect, as well as additional stream processing functionality like single message transforms, aggregations, multiplexing, enrichments, and more. It also has native support for WebAssembly (WASM) for more advanced processing.

Previously, WarpStream was just a replacement for Apache Kafka®, but with Managed Data Pipelines WarpStream can do a lot more out of the box. For example, the WarpStream Agents can now directly connect with external systems, stream data between topics, perform on-the-fly data transformation, stream data into downstream systems, and much more, all with a simple YAML configuration.

In addition to all the native features of Benthos, WarpStream Managed Data Pipelines also provide:

• A user interface in the WarpStream Console for creating and editing pipelines.
• The ability to pause and resume pipelines on demand.
• Version control and branching allow you to easily deploy changes or roll configurations backwards as needed.
• Automatic handling of SASL authentication, ACLs, AZ-aware routing, and many other WarpStream-native features.
• Control over concurrency, with distribution managed by WarpStream and controlled with a single line in your configuration.

Control versioning and deployment from within the WarpStream console

Managed Data Pipelines is the natural evolution of WarpStream’s BYOC product. In the same vein that WarpStream is a novel implementation of an open protocol with no vendor lock-in, Managed Data Pipelines is a cloud-native, BYOC-managed version of a popular open-source stream processing framework, enhanced with WarpStream’s signature data plane/control plane split. Pipeline configuration, version control, clustering, and pipeline deployment are all administered remotely using a SaaS UX, but the actual pipelines run in your cloud account, using your compute resources and your object storage buckets. Raw data never leaves your account.

To learn more about how to use Managed Data Pipelines, check out the docs, and if you have any questions, feel free to join our Slack community. You can get started with WarpStream for free, with no credit card required, and start streaming with Managed Data Pipelines in just a few minutes.