Running Kafka Locally: A Step-by-Step Guide

I am writing this because on the web there is no proper way to bootstrap kakfa locally and i spent few hours to figure out an issue !! frustrating right !

Introduction

Kafka is a powerful distributed streaming platform that is widely used for real-time data processing and messaging. To get started with Kafka without the need for a cloud-based environment, you can run it locally on your development machine. This guide will walk you through the process of setting up a local Kafka cluster using Docker Compose.

Prerequisites

• Docker installed on your system

Steps:

1. Create a Docker Compose File: Create a file named docker-compose.yml with the following content:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:2.11-1.1.1
    ports:
      - "9092:9092"
    links:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: ${HOST_IP}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_CREATE_TOPICS: "topic-test:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

KAFKA_ADVERTISED_HOST_NAME: ${HOST_IP}

This is very Important line while setting up kafka localy, this will allow local system to use IP and connect to kafka container, so get your IP quickly ! we can HOST_IP while booting up docker compose

1. Start the Kafka Cluster: Run the following command in your terminal:

  export HOST_IP=$(ifconfig | grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}" | grep -v 127.0.0.1 | awk '{ print $2 }' | cut -f2 -d: | head -n1)
docker-compose up

This will start the Zookeeper and Kafka containers in the background.

Using Kafka

Once Kafka is running, you can produce and consume messages using a Kafka client library. Here's an example using the kafkajs library:

const { Kafka } = require('kafkajs')
const ip = require('ip')

const host = process.env.HOST_IP || ip.address()
console.log(host)

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: [`${host}:9092`],
})

const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'test-group' })

const run = async () => {
  // Producing
  await producer.connect()
  await producer.send({
    topic: 'topic-test',
    messages: [
      { value: 'Hello KafkaJS user!' },
    ],
  })

  // Consuming
  await consumer.connect()
  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        value: message.value.toString(),
      })
    },
  })
}

run().catch(console.error)

Additional Tips:

• Kafka Connect: Use Kafka Connect to integrate with other data sources and sinks.
• Schema Registry: Implement a schema registry to ensure data consistency and compatibility.
• Security: Configure security measures like authentication and authorization for your Kafka cluster.

By following these steps, you can set up a local Kafka cluster and start experimenting with its capabilities.