Local Kafka cluster on Kubernetes on your PC in 5 minutes

I’m joining a Kafka related project. I want to set up a cluster as easy as I can. Confluent provides two quick starts with local and docker-compose. However, I want to use Kubernetes for hosting local Kafka cluster.

Confluent Platform Quick Start (Docker) - Confluent Platform

I create a Kafka cluster on kubernetes using Docker for Mac and Docker for Windows. It works fine.

Docker for Mac Settings

If you don’t have it, please install from here. I tested on Mac; however, Docker for Windows works might work as well.

Install Docker Desktop for Mac

Memory settings

Go to the Docker for Mac and select Preferences. Change the Memory up to 8GB+.

Memory settings

In case of Windows, you might encounter the The operation failed with error code '32788'. As my workaround, change the Hyper-V settings of the MobyLinuxVM on Hyper-V Manager. Then click Enable Dynamic Memory. Then restart docker. It worked in my environment.

Setup The Shared Drives.

Shared Drive

If you can’t share C drive, you can refer this workaround. It works for me. Create a new user and give admin role to your local machine. Then give the user permission (full control) on your C:/Users/yourname/.docker folder.

Unable to share drives · Issue #803 · docker/for-win

Enable Kubernetes

Add the check mark for the Kubernetes. It is effortless.

Enable kubernetes

Change the context

Go to the Docker for Mac and select Kubernetes, you can see the context of the Kuberentes cluster. Select the docker-for-desktop

Kubernetes Context

That’s it. You already have a Kubernetes cluster.If you don’t have a kubectl command, please refer to this page.

Install and Set Up kubectl

Install Kafka

I use helm for installing a Kafka cluster. Kafka cluster is complex and needs a lot of resources. Let them do it. If you don’t have helm command,

Helm - The Package Manager for Kubernetes.

To activate helm on your local kubernetes cluster, you need to initialize tiller.

$ helm init

I use cp-kafka helm chart.

confluentinc/cp-helm-charts

Clone that repo, then copy value.yaml file.

$ git clone https://github.com/confluentinc/cp-helm-charts.git
$ cp cp-helm-charts/values.yaml .

Then edit the values.yaml file. We need to enable external access from the cluster. Add there to the cp-kafka part.

If you not familier with the values.yaml, I put the full values.yaml file at the end of this blog post.

Then you can install the helm chart.

$ helm install --name my-confluent cp-helm-charts -f ./values.yaml

If the configuration is a success, you can see the three NodePorts are configured.

$ kubectl get svc
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP             2h
my-confluent-cp-control-center       ClusterIP   10.108.226.6     <none>        9021/TCP            2h
my-confluent-cp-kafka                ClusterIP   10.100.158.7     <none>        9092/TCP            2h
my-confluent-cp-kafka-0-nodeport     NodePort    10.96.101.240    <none>        19092:31090/TCP     2h
my-confluent-cp-kafka-1-nodeport     NodePort    10.107.131.51    <none>        19092:31091/TCP     2h
my-confluent-cp-kafka-2-nodeport     NodePort    10.108.100.161   <none>        19092:31092/TCP     2h
my-confluent-cp-kafka-connect        ClusterIP   10.105.238.212   <none>        8083/TCP            2h
my-confluent-cp-kafka-headless       ClusterIP   None             <none>        9092/TCP            2h
my-confluent-cp-kafka-rest           ClusterIP   10.107.230.111   <none>        8082/TCP            2h
my-confluent-cp-ksql-server          ClusterIP   10.98.109.66     <none>        8088/TCP            2h
my-confluent-cp-schema-registry      ClusterIP   10.111.113.199   <none>        8081/TCP            2h
my-confluent-cp-zookeeper            ClusterIP   10.99.128.38     <none>        2181/TCP            2h
my-confluent-cp-zookeeper-headless   ClusterIP   None             <none>        2888/TCP,3888/TCP   2h

Now, you can access the Kafka cluster. You can access it by ‘localhost:31090’.

$ kafkacat -b localhost:31090 -L
Metadata for all topics (from broker 0: localhost:31090/0):
 3 brokers:
  broker 0 at localhost:31090
  broker 2 at localhost:31092
  broker 1 at localhost:31091
 45 topics:
  topic "_confluent-controlcenter-5-2-0-1-group-aggregate-store-ONE_MINUTE-changelog" with 4 partitions:
    partition 0, leader 1, replicas: 1,0,2, isrs: 0,1,2
    partition 1, leader 0, replicas: 0,2,1, isrs: 0,2,1
    partition 2, leader 2, replicas: 2,1,0, isrs: 0,2,1
    partition 3, leader 1, replicas: 1,2,0, isrs: 0,1,2
       :

I’ve never tried it. However, you might use Port-Forwarding as well.

Test

You can create a kafka-client on the cluster. It helps to execute Kafka commands, e.g. create a topic.

$ cd cp-helm-chart/examples
$ kubectl apply -f kafka-client.yaml

Now you can see the pod.

$ kubectl get pods
NAME                                               READY     STATUS    RESTARTS   AGE
kafka-client                                       1/1       Running   0          2h
my-confluent-cp-control-center-67694cb78c-fqp82    1/1       Running   1          2h
my-confluent-cp-kafka-0                            2/2       Running   0          2h
my-confluent-cp-kafka-1                            2/2       Running   0          2h
my-confluent-cp-kafka-2                            2/2       Running   0          2h
my-confluent-cp-kafka-connect-b9b7db94d-95vxg      2/2       Running   1          2h
my-confluent-cp-kafka-rest-57bb8c5c7b-ngm2p        2/2       Running   1          2h
my-confluent-cp-ksql-server-6c798df75f-rh9vn       2/2       Running   0          2h
my-confluent-cp-schema-registry-75b9d5d5fd-8kjlk   2/2       Running   2          2h
my-confluent-cp-zookeeper-0                        2/2       Running   0          2h
my-confluent-cp-zookeeper-1                        2/2       Running   0          2h
my-confluent-cp-zookeeper-2                        2/2       Running   0          2h

Execute commands inside of the container. This command create a topic from the pod.

This operation is difficult for windows. In case of windows, you can use WSL. You can copy ~/.kube/conf on your windows, to the ~/.kube/conf on your WSL. Now you can access k8s from WSL.

$ kubectl exec -it kafka-client -- /bin/bash
root@kafka-client:/# cd /usr/bin
root@kafka-client:/usr/bin# kafka-topics --zookeeper my-confluent-cp-zookeeper-headless:2181 --topic my-confluent-topic --create --partitions 1 --replication-factor 1 --if-not-exists

You can send / receive using kafkacat.

terminal 1

$ kafkacat -b localhost:31090 -L -P -t my-confluent-topic
hello
confluence

terminal 2

$ kafkacat -b localhost:31090 -C -t my-confluent-topic
hello
confluence

Happy kafka!

kafkacat Utility - Confluent Platform

Reference

This blog post is an excellent article for the Kubernetes with Docker for Mac/Windows.

Accessing an application on Kubernetes in Docker

values.yaml file (full version)

Windows Memory allocation problem issue and resources.

Docker won't start, even with new installation, error code 32788 · Issue #2503 · docker/for-win

Hyper-V dynamic memory - Managing VM RAM better way

Hyper-V Failed to Change State Error Code 32788