Deploying Tigris on AWS EKS

Ovais Tariq - Oct 31 '22 - - Dev Community

This blog outlines the deployment of Tigris on an AWS managed Elastic Kubernetes Service (EKS). Future blogs we will walk through other aspects of the product setup for Tigris as a service, such as the setup of authentication. Stay tuned!

The installation will use recommended settings for redundancy, allocating more resources than a simple laptop based installation would. For more information on the laptop based installation please consult our previous blog!

If you would rather watch a video, check out the deployment in action on YouTube:

Requirements

Installation Host

The following components will need to be installed on the machine you are performing the deployment steps on:

  • Helm
  • AWS Cli

Helm will be used to install the Tigris Stack Chart:

❯ helm version
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}
Enter fullscreen mode Exit fullscreen mode

AWS Cli will be used to setup a wildcard certificate:

❯ aws --version
aws-cli/2.8.5 Python/3.10.8 Darwin/21.6.0 source/arm64 prompt/off
Enter fullscreen mode Exit fullscreen mode

EKS

Outside of the above you will need an EKS cluster with access and sufficient resources available for deployment.

❯ kubectl get nodes -L beta.kubernetes.io/instance-type
NAME                                        STATUS   ROLES    AGE   VERSION                INSTANCE-TYPE
ip-10-2-10-204.us-west-2.compute.internal   Ready    <none>   18h   v1.21.14-eks-ba74326   r6i.2xlarge
ip-10-2-10-6.us-west-2.compute.internal     Ready    <none>   6d    v1.21.14-eks-ba74326   r6i.2xlarge
ip-10-2-11-166.us-west-2.compute.internal   Ready    <none>   18h   v1.21.14-eks-ba74326   r6i.2xlarge
ip-10-2-11-169.us-west-2.compute.internal   Ready    <none>   18h   v1.21.14-eks-ba74326   r6i.2xlarge
ip-10-2-12-79.us-west-2.compute.internal    Ready    <none>   6d    v1.21.14-eks-ba74326   r6i.2xlarge
ip-10-2-13-192.us-west-2.compute.internal   Ready    <none>   6d    v1.21.14-eks-ba74326   r6i.2xlarge
Enter fullscreen mode Exit fullscreen mode

Most of the resources will be consumed by FoundationDB and TypeSense, both of which are main building blocks to Tigris.

The EKS cluster must have the following components installed:

  • AWS Load Balancer Controller
  • Cert Manager
  • EBS CSI Controller
❯ helm list -A
NAME                            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                               APP VERSION
aws-load-balancer-controller    kube-system 1           2022-10-20 11:00:30.800715 -0400 EDT    deployed    aws-load-balancer-controller-1.4.4  v2.4.3
cert-manager                    kube-system 1           2022-10-20 11:00:27.772092 -0400 EDT    deployed    cert-manager-v1.6.1                 v1.6.1
cert-manager-ca                 kube-system 1           2022-10-20 11:01:45.36529 -0400 EDT     deployed    cert-manager-ca-0.2.0               v0.1.0
external-dns                    kube-system 1           2022-10-20 11:00:25.898907 -0400 EDT    deployed    external-dns-6.7.1                  0.12.1
metrics-server                  kube-system 1           2022-10-20 11:00:26.973139 -0400 EDT    deployed    metrics-server-3.8.1                0.6.1
Enter fullscreen mode Exit fullscreen mode

Deployment

The installation deploys the following components:

  • Kubernetes Operator for FoundationDB
  • FoundationDB
  • Tigris Search (TypeSense)
  • Tigris Server

You can install the components individually or together, using the encompassing tigris-stack Helm Chart. Below I'm going to use this Chart to install Tigris.

Create Certificate for TLS

First, we need to generate a certificate for TLS. This certificate will be used on the load balancer to terminate TLS connections:

❯ aws acm request-certificate --domain-name='*.example.com'
{
    "CertificateArn": "arn:aws:acm:us-west-2:878843336588:certificate/fe257207-b117-4db0-ad6b-eef8d66308cd"
}
Enter fullscreen mode Exit fullscreen mode

Prepare For Deployment

Next, check out the deploy script repository:

❯ git clone git@github.com:tigrisdata/tigris-deploy.git
Cloning into 'tigris-deploy'...
remote: Enumerating objects: 177, done.
remote: Counting objects: 100% (97/97), done.
remote: Compressing objects: 100% (60/60), done.
remote: Total 177 (delta 43), reused 68 (delta 34), pack-reused 80
Receiving objects: 100% (177/177), 87.68 KiB | 568.00 KiB/s, done.
Resolving deltas: 100% (63/63), done.
Enter fullscreen mode Exit fullscreen mode

Navigate to the folder which contains the helm chart of tigris-stack:

❯ cd tigris-deploy/helm/tigris-stack
Enter fullscreen mode Exit fullscreen mode

Deploy Tigris Stack

As we are deploying to EKS, we are going to use an ALB as our load balancer. The host for this installation will be set to api.example.com.

⚠️ You will want to make sure that above hostname matches the domain of the wildcard certificate you created previously!

Finally, to ensure there is initial quorum for Tigris Search, we deploy it initially with a single replica:

❯ helm install tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=1

NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 18:58:53 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
Enter fullscreen mode Exit fullscreen mode

That's it, your Tigris deployment should be now on its way coming up!

Validate Deployment

Generally speaking, there are two high level checkboxes to check. First, we should ensure that all the resources were scheduled and are available and running. Second, we will want to make sure that Tigris API is accessible using the Ingress resource allocated. These steps are expanded upon below.

Resources Validation

Allow the resources such as FoundationDB a couple minutes to initialize. In a Production-ready installation FoundationDB would allocate significant resources. You will want to make sure that the FDB Pods were able to be scheduled.

Tigris will not enter the Running state until FoundationDB becomes fully functional. It might take a couple minutes for FoundationDB to become unavailable, even when all of its Pods appear to be in the Running state.

You can validate if the FDB key-value store is available using fdbcli:

tigris@tigris-server-58ccd7bb9f-czcbb:/server$ fdbcli -C /mnt/fdb-config-volume/cluster-file
Using cluster file `/mnt/fdb-config-volume/cluster-file'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb>
Enter fullscreen mode Exit fullscreen mode

Look for the message "The database is available" in fdbcli's output.

Final Overview

Unless there were additional customizations, your output should be similar to below:

❯ kubectl get all
NAME                                             READY   STATUS    RESTARTS   AGE
pod/fdb-cluster-log-1                            2/2     Running   0          5m53s
pod/fdb-cluster-log-2                            2/2     Running   0          5m53s
pod/fdb-cluster-log-3                            2/2     Running   0          5m53s
pod/fdb-cluster-log-4                            2/2     Running   0          5m53s
pod/fdb-cluster-log-5                            2/2     Running   0          5m53s
pod/fdb-cluster-stateless-1                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-10                     2/2     Running   0          5m53s
pod/fdb-cluster-stateless-2                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-3                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-4                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-5                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-6                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-7                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-8                      2/2     Running   0          5m54s
pod/fdb-cluster-stateless-9                      2/2     Running   0          5m53s
pod/fdb-cluster-storage-1                        2/2     Running   0          5m54s
pod/fdb-cluster-storage-2                        2/2     Running   0          5m54s
pod/fdb-cluster-storage-3                        2/2     Running   0          5m54s
pod/fdb-cluster-storage-4                        2/2     Running   0          5m54s
pod/fdb-cluster-storage-5                        2/2     Running   0          5m54s
pod/tigris-search-0                              2/2     Running   1          6m49s
pod/tigris-server-58ccd7bb9f-czcbb               1/1     Running   4          6m49s
pod/tigris-server-58ccd7bb9f-ngjk5               1/1     Running   4          6m49s
pod/tigris-server-58ccd7bb9f-rnbxb               1/1     Running   4          6m49s
pod/tigris-stack-fdb-operator-5d9dbc4c9d-ptlng   1/1     Running   0          6m49s

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes        ClusterIP   172.20.0.1       <none>        443/TCP        5d8h
service/tigris-grpc       NodePort    172.20.60.127    <none>        80:30440/TCP   6m50s
service/tigris-headless   ClusterIP   None             <none>        8080/TCP       6m50s
service/tigris-http       NodePort    172.20.82.191    <none>        80:30675/TCP   6m50s
service/tigris-search     NodePort    172.20.130.194   <none>        80:31720/TCP   6m50s
service/ts                ClusterIP   None             <none>        8108/TCP       6m50s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tigris-server               3/3     3            3           6m50s
deployment.apps/tigris-stack-fdb-operator   1/1     1            1           6m50s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/tigris-server-58ccd7bb9f               3         3         3       6m50s
replicaset.apps/tigris-stack-fdb-operator-5d9dbc4c9d   1         1         1       6m50s

NAME                             READY   AGE
statefulset.apps/tigris-search   1/1     6m50s
Enter fullscreen mode Exit fullscreen mode

Ingress Validation

The most EKS installation specific piece to a Tigris installation is generally the load balancer and related resources.

The installation will create an annotated Ingress resource:

❯ kubectl get ingress
NAME            CLASS    HOSTS   ADDRESS   PORTS   AGE
tigris-server   <none>   *                 80      3m25s
Enter fullscreen mode Exit fullscreen mode
❯ kubectl get ingress tigris-server -o yaml | grep -A17 annotations:
  annotations:
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/conditions.tigris-grpc: |
      [{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
    alb.ingress.kubernetes.io/group.name: tigris-server-lb
    alb.ingress.kubernetes.io/healthcheck-path: /v1/health
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/load-balancer-attributes: routing.http.drop_invalid_header_fields.enabled=true
    alb.ingress.kubernetes.io/load-balancer-name: tigris-server-lb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
    alb.ingress.kubernetes.io/tags: service=tigris-server
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.app_cookie.duration_seconds=10,stickiness.type=app_cookie,stickiness.app_cookie.cookie_name=Tigris-Tx-Id
    alb.ingress.kubernetes.io/target-type: ip
    external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
    kubernetes.io/ingress.class: alb
    meta.helm.sh/release-name: tigris-stack
    meta.helm.sh/release-namespace: default
Enter fullscreen mode Exit fullscreen mode

AWS Load Balancer Controller will create an ALB based on above annotation:

❯ aws elbv2 describe-load-balancers --names tigris-server-lb | grep -i dnsname
            "DNSName": "tigris-server-lb-<redacted>.us-west-2.elb.amazonaws.com",
Enter fullscreen mode Exit fullscreen mode

Make sure that your load balancer is healthy and operational before you proceed!

Preparing For Production

There is one last step that is required for a proper, production-ready installation.

In order to ensure there is proper redundancy under Tigris Search, you will want to increase the number of replicas to an odd number of replicas post initial installation.

An odd number of replicas is required to ensure that quorum can be reached. An even number of replicas could end up being without a tie breaker during network partitioning, where both partitions may end up with the same number of replicas.

Below command will increase the number of Tigris Search replicas to 5:

❯ helm upgrade tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=5
Release "tigris-stack" has been upgraded. Happy Helming!
NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 19:13:33 2022
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
Enter fullscreen mode Exit fullscreen mode

We recommend using at least 3 replicas for Production installations with 5 being recommended for load handling purposes. This is the default setting in the tigris-stack Chart that is only reduced during install, to ensure quorum can be achieved during initialization.

You can verify that there are now 5 replicas running with a simple kubectl command:

❯ kubectl get pods | grep tigris
tigris-search-0                              2/2     Running   1          17m
tigris-search-1                              2/2     Running   0          2m39s
tigris-search-2                              2/2     Running   0          2m39s
tigris-search-3                              2/2     Running   0          2m39s
tigris-search-4                              2/2     Running   0          2m39s
tigris-server-58ccd7bb9f-czcbb               1/1     Running   4          17m
tigris-server-58ccd7bb9f-ngjk5               1/1     Running   4          17m
tigris-server-58ccd7bb9f-rnbxb               1/1     Running   4          17m
tigris-stack-fdb-operator-5d9dbc4c9d-ptlng   1/1     Running   0          17m
Enter fullscreen mode Exit fullscreen mode

Final Thoughts

In this blog we have covered the deployment of Tigris to one of the most common Managed Kubernetes platforms available in the Cloud. As you can see from above the deployment process is fairly easy and straightforward on EKS.

If you liked our blog make sure to follow us as next week we are going to cover the deployment of Tigris on Google Cloud Platform!

Enjoy using Tigris as much as we enjoy building it!!!


Tigris is the data platform built for developers! Use it as a scalable, ACID transactional, real-time backend for your serverless applications. Build data-rich features without worrying about slow queries or missing indexes. Seamlessly implement search within your applications with its embedded search engine. Connect serverless functions with its event streams to build highly responsive applications that scale automatically.

Sign up for the beta

Get early access and try out Tigris for your next application. Join our Slack or Discord community to ask any questions you might have.

. . . . . . . . . . . . . . . . . . . . . .