Ovais TariqThis blog outlines the deployment of Tigris on an AWS managed Elastic Kubernetes Service (EKS). Future blogs we will walk through other aspects of the product setup for Tigris as a service, such as the setup of authentication. Stay tuned!
The installation will use recommended settings for redundancy, allocating more resources than a simple laptop based installation would. For more information on the laptop based installation please consult our previous blog!
If you would rather watch a video, check out the deployment in action on YouTube:
Requirements
Installation Host
The following components will need to be installed on the machine you are performing the deployment steps on:
- Helm
- AWS Cli
Helm will be used to install the Tigris Stack Chart:
❯ helm version
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}
AWS Cli will be used to setup a wildcard certificate:
❯ aws --version
aws-cli/2.8.5 Python/3.10.8 Darwin/21.6.0 source/arm64 prompt/off
EKS
Outside of the above you will need an EKS cluster with access and sufficient resources available for deployment.
❯ kubectl get nodes -L beta.kubernetes.io/instance-type
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE
ip-10-2-10-204.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-10-6.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-11-166.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-11-169.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-12-79.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-13-192.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge
Most of the resources will be consumed by FoundationDB and TypeSense, both of which are main building blocks to Tigris.
The EKS cluster must have the following components installed:
- AWS Load Balancer Controller
- Cert Manager
- EBS CSI Controller
❯ helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
aws-load-balancer-controller kube-system 1 2022-10-20 11:00:30.800715 -0400 EDT deployed aws-load-balancer-controller-1.4.4 v2.4.3
cert-manager kube-system 1 2022-10-20 11:00:27.772092 -0400 EDT deployed cert-manager-v1.6.1 v1.6.1
cert-manager-ca kube-system 1 2022-10-20 11:01:45.36529 -0400 EDT deployed cert-manager-ca-0.2.0 v0.1.0
external-dns kube-system 1 2022-10-20 11:00:25.898907 -0400 EDT deployed external-dns-6.7.1 0.12.1
metrics-server kube-system 1 2022-10-20 11:00:26.973139 -0400 EDT deployed metrics-server-3.8.1 0.6.1
Deployment
The installation deploys the following components:
- Kubernetes Operator for FoundationDB
- FoundationDB
- Tigris Search (TypeSense)
- Tigris Server
You can install the components individually or together, using the encompassing tigris-stack Helm Chart. Below I'm going to use this Chart to install Tigris.
Create Certificate for TLS
First, we need to generate a certificate for TLS. This certificate will be used on the load balancer to terminate TLS connections:
❯ aws acm request-certificate --domain-name='*.example.com'
{
"CertificateArn": "arn:aws:acm:us-west-2:878843336588:certificate/fe257207-b117-4db0-ad6b-eef8d66308cd"
}
Prepare For Deployment
Next, check out the deploy script repository:
❯ git clone git@github.com:tigrisdata/tigris-deploy.git
Cloning into 'tigris-deploy'...
remote: Enumerating objects: 177, done.
remote: Counting objects: 100% (97/97), done.
remote: Compressing objects: 100% (60/60), done.
remote: Total 177 (delta 43), reused 68 (delta 34), pack-reused 80
Receiving objects: 100% (177/177), 87.68 KiB | 568.00 KiB/s, done.
Resolving deltas: 100% (63/63), done.
Navigate to the folder which contains the helm chart of tigris-stack:
❯ cd tigris-deploy/helm/tigris-stack
Deploy Tigris Stack
As we are deploying to EKS, we are going to use an ALB as our load balancer. The host for this installation will be set to api.example.com.
⚠️ You will want to make sure that above hostname matches the domain of the wildcard certificate you created previously!
Finally, to ensure there is initial quorum for Tigris Search, we deploy it initially with a single replica:
❯ helm install tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=1
NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 18:58:53 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
That's it, your Tigris deployment should be now on its way coming up!
Validate Deployment
Generally speaking, there are two high level checkboxes to check. First, we should ensure that all the resources were scheduled and are available and running. Second, we will want to make sure that Tigris API is accessible using the Ingress resource allocated. These steps are expanded upon below.
Resources Validation
Allow the resources such as FoundationDB a couple minutes to initialize. In a Production-ready installation FoundationDB would allocate significant resources. You will want to make sure that the FDB Pods were able to be scheduled.
Tigris will not enter the Running state until FoundationDB becomes fully functional. It might take a couple minutes for FoundationDB to become unavailable, even when all of its Pods appear to be in the Running state.
You can validate if the FDB key-value store is available using fdbcli:
tigris@tigris-server-58ccd7bb9f-czcbb:/server$ fdbcli -C /mnt/fdb-config-volume/cluster-file
Using cluster file `/mnt/fdb-config-volume/cluster-file'.
The database is available.
Welcome to the fdbcli. For help, type `help'.
fdb>
Look for the message "The database is available" in fdbcli's output.
Final Overview
Unless there were additional customizations, your output should be similar to below:
❯ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/fdb-cluster-log-1 2/2 Running 0 5m53s
pod/fdb-cluster-log-2 2/2 Running 0 5m53s
pod/fdb-cluster-log-3 2/2 Running 0 5m53s
pod/fdb-cluster-log-4 2/2 Running 0 5m53s
pod/fdb-cluster-log-5 2/2 Running 0 5m53s
pod/fdb-cluster-stateless-1 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-10 2/2 Running 0 5m53s
pod/fdb-cluster-stateless-2 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-3 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-4 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-5 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-6 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-7 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-8 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-9 2/2 Running 0 5m53s
pod/fdb-cluster-storage-1 2/2 Running 0 5m54s
pod/fdb-cluster-storage-2 2/2 Running 0 5m54s
pod/fdb-cluster-storage-3 2/2 Running 0 5m54s
pod/fdb-cluster-storage-4 2/2 Running 0 5m54s
pod/fdb-cluster-storage-5 2/2 Running 0 5m54s
pod/tigris-search-0 2/2 Running 1 6m49s
pod/tigris-server-58ccd7bb9f-czcbb 1/1 Running 4 6m49s
pod/tigris-server-58ccd7bb9f-ngjk5 1/1 Running 4 6m49s
pod/tigris-server-58ccd7bb9f-rnbxb 1/1 Running 4 6m49s
pod/tigris-stack-fdb-operator-5d9dbc4c9d-ptlng 1/1 Running 0 6m49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 5d8h
service/tigris-grpc NodePort 172.20.60.127 <none> 80:30440/TCP 6m50s
service/tigris-headless ClusterIP None <none> 8080/TCP 6m50s
service/tigris-http NodePort 172.20.82.191 <none> 80:30675/TCP 6m50s
service/tigris-search NodePort 172.20.130.194 <none> 80:31720/TCP 6m50s
service/ts ClusterIP None <none> 8108/TCP 6m50s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tigris-server 3/3 3 3 6m50s
deployment.apps/tigris-stack-fdb-operator 1/1 1 1 6m50s
NAME DESIRED CURRENT READY AGE
replicaset.apps/tigris-server-58ccd7bb9f 3 3 3 6m50s
replicaset.apps/tigris-stack-fdb-operator-5d9dbc4c9d 1 1 1 6m50s
NAME READY AGE
statefulset.apps/tigris-search 1/1 6m50s
Ingress Validation
The most EKS installation specific piece to a Tigris installation is generally the load balancer and related resources.
The installation will create an annotated Ingress resource:
❯ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tigris-server <none> * 80 3m25s
❯ kubectl get ingress tigris-server -o yaml | grep -A17 annotations:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/conditions.tigris-grpc: |
[{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
alb.ingress.kubernetes.io/group.name: tigris-server-lb
alb.ingress.kubernetes.io/healthcheck-path: /v1/health
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http.drop_invalid_header_fields.enabled=true
alb.ingress.kubernetes.io/load-balancer-name: tigris-server-lb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
alb.ingress.kubernetes.io/tags: service=tigris-server
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.app_cookie.duration_seconds=10,stickiness.type=app_cookie,stickiness.app_cookie.cookie_name=Tigris-Tx-Id
alb.ingress.kubernetes.io/target-type: ip
external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: tigris-stack
meta.helm.sh/release-namespace: default
AWS Load Balancer Controller will create an ALB based on above annotation:
❯ aws elbv2 describe-load-balancers --names tigris-server-lb | grep -i dnsname
"DNSName": "tigris-server-lb-<redacted>.us-west-2.elb.amazonaws.com",
Make sure that your load balancer is healthy and operational before you proceed!
Preparing For Production
There is one last step that is required for a proper, production-ready installation.
In order to ensure there is proper redundancy under Tigris Search, you will want to increase the number of replicas to an odd number of replicas post initial installation.
An odd number of replicas is required to ensure that quorum can be reached. An even number of replicas could end up being without a tie breaker during network partitioning, where both partitions may end up with the same number of replicas.
Below command will increase the number of Tigris Search replicas to 5:
❯ helm upgrade tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=5
Release "tigris-stack" has been upgraded. Happy Helming!
NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 19:13:33 2022
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
We recommend using at least 3 replicas for Production installations with 5 being recommended for load handling purposes. This is the default setting in the tigris-stack Chart that is only reduced during install, to ensure quorum can be achieved during initialization.
You can verify that there are now 5 replicas running with a simple kubectl command:
❯ kubectl get pods | grep tigris
tigris-search-0 2/2 Running 1 17m
tigris-search-1 2/2 Running 0 2m39s
tigris-search-2 2/2 Running 0 2m39s
tigris-search-3 2/2 Running 0 2m39s
tigris-search-4 2/2 Running 0 2m39s
tigris-server-58ccd7bb9f-czcbb 1/1 Running 4 17m
tigris-server-58ccd7bb9f-ngjk5 1/1 Running 4 17m
tigris-server-58ccd7bb9f-rnbxb 1/1 Running 4 17m
tigris-stack-fdb-operator-5d9dbc4c9d-ptlng 1/1 Running 0 17m
Final Thoughts
In this blog we have covered the deployment of Tigris to one of the most common Managed Kubernetes platforms available in the Cloud. As you can see from above the deployment process is fairly easy and straightforward on EKS.
If you liked our blog make sure to follow us as next week we are going to cover the deployment of Tigris on Google Cloud Platform!
Enjoy using Tigris as much as we enjoy building it!!!
Tigris is the data platform built for developers! Use it as a scalable, ACID transactional, real-time backend for your serverless applications. Build data-rich features without worrying about slow queries or missing indexes. Seamlessly implement search within your applications with its embedded search engine. Connect serverless functions with its event streams to build highly responsive applications that scale automatically.
Get early access and try out Tigris for your next application. Join our Slack or Discord community to ask any questions you might have.
