Taavi RehemägiDashbird got an update, and you can now monitor the OpenSearch clusters you set up with Amazon OpenSearch Service. But what does this even mean? Many people don't even know what OpenSearch is! Wasn't there an Elasticsearch service before? So, let's have some explanation before we check out Dashbirds OpenSearch support.
Elastic and AWS Disagreed on Fair Use of Elasticsearch
After a heated argument between Elastic, the main contributor to the Elasticsearch (ES) project, and AWS. Elastic then changed the ES license to prohibit AWS from offering their Amazon Elasticsearch Service. Since the ES versions previous to that change were still under the old Apache 2.0 license, AWS created a fork from these and called it OpenSearch (OS).
To put it simply: Two companies disagreed on some software they implemented, and each one went on with their vision for that project. So, OS is merely a fork of ES, and these two open-source projects will probably diverge in the future in terms of compatibility and features.
In terms of managed search services, AWS deprecated Amazon Elasticsearch Service, and Amazon OpenSearch Service (AOS) is the way to go.
What is OpenSearch?
OS is a search engine, just like Google or Bing, but for your own data. It will index your data and make it searchable via an HTTP interface.
You can download OS and set it up on whatever computer you like. Install it on your local machine, set up an EC2 instance on AWS and install it there, or let AWS help you run it with the AOS.
Get the full size infographic from here.
OpenSearch vs. Elasticsearch
They are pretty similar, but each one is on a different trajectory. Some developers aren't happy with AWS treating Elastic the way they did. Others say switching to a less permissive license isn't the open-source way, and while they don't like how AWS might have treated Elastic, they think the Apache 2.0 license is the way to go if things should stay open.
If you want to use a managed, ES-like system on AWS, you either have to use older versions of ES or OS.
What is Amazon OpenSearch Service?
First, while AOS is a managed service, it's not serverless; you have to pay for idle servers.
But what is AOS actually?
AOS is the service on AWS that manages OS deployments for you. Like RDS is for relational databases, EKS is for Kubernetes clusters, and Elasticsearch Service is for ES, so is AOS for OS.
While AOS is an excellent help to get OS running in the cloud, ES and, in turn, OS is notoriously hard to set up and scale. So hard that no serverless versions of them exist.
While AOS isn't that easy to use, it's still simpler than managing your OS instances manually. AOS helps with scaling clusters, deploying VMs, and managing the underlying network and storage for you. It will also install bug fixes.
If you need a cloud-based search engine for your project, you might not get around messing with OS and AOS.
Amazon OpenSearch Service vs. Cloudwatch Log Insights
When you deploy your infrastructure to the cloud, chances are most of the data you generate is operational logs. If you want to ease the access of these logs for the ops team, you might be inclined to pump them into AOS. But this can get costly.
As mentioned above, AOS isn't a serverless system, so you pay for every hour it's deployed, even if you don't use it. This unused capacity could lead to a massive bill for nothing regarding operational monitoring data.
Cloudwatch Log Insights is a managed service on top of Cloudwatch logs and, in turn, optimized for this exact use-case. It scales automatically, and you only have to pay for the data you analyze, so it can be considered a serverless service.
AOS is a more low-level service you use if you need maximum flexibility. It costs more and isn't easy to scale, but it gives you all the freedom you need for a customized search engine.
Amazon OpenSearch Service in Serverless Architectures
As explained before, AOS isn't a serverless service. It can help with scaling, but it doesn't go down to zero, so you will pay for the underlying EC2 instances idling.
This makes integrating AOS into your architecture a hard choice. But a hard choice between which options?
If money is tight, or you believe that a system should scale to zero for whatever reason, you have to ask yourself what your search requirements are.
AOS is the nuclear option here; it's a Lucene-based distributed search engine that finds everything you indexed with Google-like features like fuzzy matching and such.
But if your search requirements aren't that strong, you might get away with DynamoDB's query features. Your results may vary and won't be on OpenSearch levels, but it might be just enough for a simple search feature. But most importantly, you don't pay for idle EC2 instances.
Setting Up OpenSearch on AWS
As said before, there are different methods of setting up OS on AWS, but we will go with the easiest here. Using the CDK, we will set up an AOS domain in just a few lines of code.
Keep in mind this example taken from the CDK docs will deploy a production-ready OS cluster containing 25 m3.medium instances, which can get expensive quickly.
const aosDomain = new opensearch.Domain(this, 'AosDomain', {
version: opensearch.EngineVersion.OPENSEARCH_1_0,
capacity: {masterNodes: 5, dataNodes: 20},
ebs: {volumeSize: 20},
zoneAwareness: {availabilityZoneCount: 3},
logging: {
slowSearchLogEnabled: true,
appLogEnabled: true,
slowIndexLogEnabled: true,
},
});
Coming from ES, you might have heard about a cluster and how it's made up of one or more nodes. An AOS domain is such a cluster, plus all the things around running such a cluster that AOS takes care of for you. Like networking, updates, resource allocation, etc.
We must use the correct engine version to get an OS cluster because AOS also supports legacy ES clusters.
The nodes are EC2 instances of the default instance type m3.medium.search.
We use five master nodes, which means there will be one master and five master-eligible nodes which can take the place of the master if something goes wrong.
We also use twenty data nodes, which will be used to store all the index shards (i.e., the actual data) of our cluster. Each of these data nodes will get a 20 GiB elastic block storage.
The zone awareness makes sure AOS will deploy our 25 EC2 instances across three availability zones. This way, the cluster won't go down if one zone becomes unavailable.
The logging config makes sure that queries and indices that perform poorly are logged so that when the cluster runs for some time, you get a feeling for the bottlenecks.
OpenSearch Monitoring with Dashbird
With the newest update to Dashbird, all AOS domains will be automatically picked up in your Dashbird inventory.
This update gives you a dashboard out of the box, with all the health metrics related to your domains. Including, but not limited to, the utilization of the different node types and the query latency.
While this is an excellent start to assess the status of your existing clusters in the context of their history, more interesting are the well-architected Dashbird insights.
The insights are periodical checks based on the AWS Well-Architected Framework. This means they test your AOS domains for best practices and notify you if you're not following them.
The insights also include some recurring checks for the availability and usage of your AOS domains. After all, you have to pay for them if they're deployed, and if they are never used, you might as well decommission them to save a few dollars.
Summary
OS is the search engine software that AWS will use from now on. If you want to stay with ES, you either have to use an older version or deploy your cluster manually with EC2, ECS, or EKS.
But rest assured that Dashbird got your back with that new ES fork. So, you will stay on top of things with monitoring metrics and well-architected insights for your AOS domains.
Further reading:
AWS Elastic Load Balancing from a Serverless perspective
