The Ultimate Comprehensive Guide to Setting Up Self-Hosted Airbyte on EKS Using Karpenter

Luqman Bello - Sep 17 - - Dev Community

Table of Contents

  1. Introduction
  2. Prerequisites
  3. Setting Up EKS with Karpenter
  4. Deploying Airbyte on EKS
  5. Upgrading Airbyte OSS Version
  6. Monitoring and Maintenance
  7. Troubleshooting Common Errors

Introduction

Airbyte is an open-source data integration platform that enables you to consolidate your data from various sources to data warehouses, lakes, and databases. This 2024 guide walks you through setting up a production-ready, self-hosted Airbyte instance on Amazon EKS (Elastic Kubernetes Service) using Karpenter for dynamic node provisioning. The setup includes best practices for security, scalability, and maintenance.

Prerequisites

  • AWS Account: An active AWS account with appropriate permissions
  • AWS CLI: Version 2.13.0 or later
  • kubectl: Version 1.28 or later
  • Helm: Version 3.12 or later
  • eksctl: Version 0.163.0 or later
  • Basic knowledge: Kubernetes, AWS, and command-line operations

Setting Up EKS with Karpenter

Creating an EKS Cluster

Create a modern EKS cluster using eksctl with the latest supported version:

eksctl create cluster \
  --name airbyte-cluster \
  --region us-west-2 \
  --version 1.28 \
  --with-oidc \
  --zones us-west-2a,us-west-2b \
  --nodegroup-name standard-workers \
  --node-type t3.large \
  --nodes 2 \
  --nodes-min 1 \
  --nodes-max 4 \
  --managed \
  --addons aws-ebs-csi-driver,vpc-cni,coredns,kube-proxy \
  --node-volume-size 100 \
  --external-dns-access
Enter fullscreen mode Exit fullscreen mode

Key changes from previous versions:

  • Updated to EKS 1.28 (latest stable)
  • Added essential EKS add-ons
  • Increased default node size to t3.large for better performance
  • Added larger root volume for containers
  • Enabled external DNS access for better integration

Installing Karpenter

  1. Create AWS IAM Resources for Karpenter

Using the latest Karpenter IAM setup:

   export CLUSTER_NAME="airbyte-cluster"
   export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
   export AWS_REGION="us-west-2"

   curl -fsSL https://karpenter.sh/v0.32.0/getting-started/getting-started-with-eksctl/cloudformation.yaml  > cloudformation.yaml

   aws cloudformation deploy \
     --stack-name "karpenter-${CLUSTER_NAME}" \
     --template-file cloudformation.yaml \
     --capabilities CAPABILITY_NAMED_IAM \
     --parameter-overrides "ClusterName=${CLUSTER_NAME}"
Enter fullscreen mode Exit fullscreen mode
  1. Install Karpenter Controller 
   helm repo add karpenter https://charts.karpenter.sh
   helm repo update

   helm install karpenter karpenter/karpenter \
     --namespace karpenter --create-namespace \
     --version v0.32.0 \
     --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::${AWS_ACCOUNT_ID}:role/karpenter-controller-${CLUSTER_NAME} \
     --set settings.aws.clusterName=${CLUSTER_NAME} \
     --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
     --set settings.aws.interruptionQueueName=karpenter-${CLUSTER_NAME} \
     --set controller.resources.requests.cpu=1 \
     --set controller.resources.requests.memory=1Gi \
     --set controller.resources.limits.cpu=1 \
     --set controller.resources.limits.memory=1Gi
Enter fullscreen mode Exit fullscreen mode
  1. Create a Modern Provisioner

Create provisioner.yaml:

   apiVersion: karpenter.sh/v1beta1
   kind: NodePool
   metadata:
     name: default
   spec:
     template:
       spec:
         requirements:
           - key: "kubernetes.io/arch"
             operator: In
             values: ["amd64"]
           - key: "kubernetes.io/os"
             operator: In
             values: ["linux"]
           - key: "karpenter.sh/capacity-type"
             operator: In
             values: ["on-demand"]
           - key: "node.kubernetes.io/instance-type"
             operator: In
             values: ["t3.large", "t3.xlarge", "m5.large", "m5.xlarge"]
         nodeClassRef:
           name: default
     limits:
       resources:
         cpu: 100
         memory: 100Gi
     disruption:
       consolidationPolicy: WhenUnderutilized
       consolidateAfter: 30s
     weight: 10
---
apiVersion: karpenter.sh/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  roleClassName: default
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}
Enter fullscreen mode Exit fullscreen mode

Apply the configuration:

   kubectl apply -f provisioner.yaml
Enter fullscreen mode Exit fullscreen mode

Deploying Airbyte on EKS

Configuring Airbyte Resources

  1. Create Namespace and Storage Class 
   kubectl create namespace airbyte

   cat <<EOF | kubectl apply -f -
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: airbyte-storage
   provisioner: ebs.csi.aws.com
   parameters:
     type: gp3
     encrypted: "true"
   reclaimPolicy: Retain
   allowVolumeExpansion: true
   volumeBindingMode: WaitForFirstConsumer
   EOF
Enter fullscreen mode Exit fullscreen mode
  1. Set Up Persistent Storage 
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: airbyte-pvc
     namespace: airbyte
   spec:
     storageClassName: airbyte-storage
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 50Gi
Enter fullscreen mode Exit fullscreen mode

Deploying Airbyte

  1. Add Airbyte Helm Repository 
   helm repo add airbyte https://airbytehq.github.io/helm-charts
   helm repo update
Enter fullscreen mode Exit fullscreen mode
  1. Create Custom Values File

Create airbyte-values.yaml:

   global:
     serviceAccountName: airbyte-admin
     database:
       secretName: airbyte-db-secrets
     logs:
      storage:
        enabled: true

   webapp:
     resources:
       limits:
         cpu: "1"
         memory: 2Gi
       requests:
         cpu: "0.5"
         memory: 1Gi

   server:
     resources:
       limits:
         cpu: "2"
         memory: 4Gi
       requests:
         cpu: "1"
         memory: 2Gi

   worker:
     resources:
       limits:
         cpu: "2"
         memory: 4Gi
       requests:
         cpu: "1"
         memory: 2Gi

   persistence:
     enabled: true
     storageClass: airbyte-storage
     size: 50Gi

   metrics:
     enabled: true
     serviceMonitor:
       enabled: true
Enter fullscreen mode Exit fullscreen mode
  1. Install Airbyte 
   helm install airbyte airbyte/airbyte \
     --namespace airbyte \
     --values airbyte-values.yaml \
     --version 0.50.0
Enter fullscreen mode Exit fullscreen mode
  1. Verify Deployment 
   kubectl get pods -n airbyte
   kubectl get svc -n airbyte
Enter fullscreen mode Exit fullscreen mode

Upgrading Airbyte OSS Version

  1. Backup Before Upgrade 
   # Backup using Velero or similar tool
   velero backup create airbyte-backup \
     --include-namespaces airbyte
Enter fullscreen mode Exit fullscreen mode
  1. Update Helm Repository 
   helm repo update
   helm search repo airbyte/airbyte --versions
Enter fullscreen mode Exit fullscreen mode
  1. Upgrade Airbyte 
   helm upgrade airbyte airbyte/airbyte \
     --namespace airbyte \
     --values airbyte-values.yaml \
     --version <NEW_VERSION> \
     --timeout 15m
Enter fullscreen mode Exit fullscreen mode

Monitoring and Maintenance

  1. Set Up Monitoring 
   # Install Prometheus and Grafana
   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

   helm install prometheus prometheus-community/kube-prometheus-stack \
     --namespace monitoring \
     --create-namespace
Enter fullscreen mode Exit fullscreen mode
  1. Configure Log Aggregation

Use AWS CloudWatch Logs:

   kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/cloudwatch-logs/cloudwatch-logs-configmap.yaml
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Errors

Pod Scheduling Issues

  • Symptom: Pods stuck in Pending state
  • Solution: Check Karpenter logs and node pool configuration: 
  kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -f
Enter fullscreen mode Exit fullscreen mode

Database Connection Issues

  • Symptom: Database connection errors in Airbyte logs
  • Solution: Verify secrets and connection strings: 
  kubectl get secrets -n airbyte
  kubectl logs -n airbyte deployment/airbyte-server
Enter fullscreen mode Exit fullscreen mode

Resource Constraints

  • Symptom: OOMKilled or CPU throttling
  • Solution: Adjust resource limits in values.yaml: 
  resources:
    limits:
      memory: "4Gi"
      cpu: "2"
    requests:
      memory: "2Gi"
      cpu: "1"
Enter fullscreen mode Exit fullscreen mode

Karpenter Node Provisioning Issues

  • Symptom: Nodes not being provisioned
  • Solution: Check NodePool and EC2NodeClass configurations: 
  kubectl describe nodepool default
  kubectl describe ec2nodeclass default
Enter fullscreen mode Exit fullscreen mode

This guide provides a robust foundation for running Airbyte on EKS with Karpenter. Remember to regularly check for updates and security patches for all components. For production environments, consider implementing additional security measures and backup strategies.

For questions, issues, or suggestions, please leave a comment below!

Image
Responsible For The Asbestosis Settlement Amounts Budget? 12 Top Ways To Spend Your Money
Image
How To Design And Create Successful Sliding Door Hook Lock With Key Instructions For Homeschoolers From Home
Image
مطالعه در خانه drazingazor دکتر آذین گازر 2024.10.28
Image
Most Useful Jenkins Plugins and Their Use Cases

explainlikeimfive, docker, devops, jenkins
Image
12 Companies Are Leading The Way In Asbestos Lawsuit Settlements
Image
Transform Your Operations with OpenTeQ’s Advanced NetSuite Developer Services
Image
surviving coding bootcamp: lessons learned

learning, bootcamp, beginners, career
Image
20 JavaScript Tricks Every Developer Must Know 🚀

webdev, javascript, discuss, beginners
Image
Top Alternatives To The Zsh Ls Command

community, webdev, beginners, programming
Image
Buy Google 5 Star Reviews Best 10 site

css, typescript, learning, discuss
Image
Power Automate - Expressions

powerautomate, powerplatform, nocode, rpa
Image
CommuneNet

devchallenge, wixstudiochallenge, webdev, javascript
Image
IT Market Trends for 2025: Staying Ahead in a Rapidly Evolving Industry

career, interview, beginners
Image
The Complete List Of Mesothelioma Asbestos Lawyers Dos And Don'ts
Image
Why Custom Software is Essential for Scaling Your Business in 2024

customsoftware, softwaredevelopment, businessgrowth, aristonit
Image
Buy GitHub account Best 10 site

webdev, javascript, beginners, programming
Image
Hire Shopify Hydrogen Experts for Headless Commerce Development

webdev, productivity, programming, learning
Image
CodeCatalyst : One place for hackathons.

devchallenge, wixstudiochallenge, webdev, javascript
Image
𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗖𝗹𝗮𝘀𝘀𝗲𝘀 𝗮𝗻𝗱 𝗢𝗯𝗷𝗲𝗰𝘁𝘀 𝗶𝗻 𝗖#: 𝗧𝗵𝗲 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗕𝗹𝗼𝗰𝗸𝘀 𝗼𝗳 𝗢𝗢𝗣
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .