Apache Kafka Multi Broker Cluster with Multi Node Zookeeper

Apache Kafka Multi Broker Cluster with Multi Node Zookeeper

Zookeeper cluster called as zookeeper ensemble (An ensemble contains more than one zookeeper nodes)

Concept of Quorum: Ceil (N/2)

Number of Nodes    |    Nodes for Quorum
--------------------------------------------------------------------
          3                                          2
          5                                          3
          7                                          4

if we have 3 zookeeper nodes atleast 2 nodes should up and running

Odd number of Zookeeper nodes recommended (In propotion odd number of nodes are more fault tollerent)
Ex : if we have 4 zookeeper nodes atleast 3 nodes should up and running, if we have 5 nodes atleast 3 nodes should up and running.

Edit configuration file

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/zookeeper.properties

tickTime=2000
Unit for measurements in milliseconds used by zookeeper used to measure 'heart-beats' and 'time-outs'
initLimit=5
Amount of time in ticks which zookeeper follower takes to connect to zookeeper leader initially when a cluster is started
syncLimit=2
Amount of time in ticks which allows for a followr to sync with a leader

We have 3 <servers>=<hostname>:<port for node to node communication in a Quorum (connect follower to leader)>:<port to elect new leader>
server.1=localhost:2666:3666
server.2=localhost:2667:3667
server.3=localhost:2668:3668

Copy and create multiple config files

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp config/zookeeper.properties config/zookeeper1.properties
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp config/zookeeper.properties config/zookeeper2.properties

Change 'dataDir=/tmp/zookeeper1', 'clientPort=2182'

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/zookeeper1.properties

Change 'dataDir=/tmp/zookeeper2', 'clientPort=2183'

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/zookeeper2.properties

Create datadir

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ vi  /tmp/zookeeper/myid
add value '1'

Copy and create more directory in same location

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp -R /tmp/zookeeper /tmp/zookeeper1
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp -R /tmp/zookeeper /tmp/zookeeper2

Edit the value '2', '3'

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano /tmp/zookeeper1/myid
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano /tmp/zookeeper2/myid

Start Zookeeper nodes

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/zookeeper-server-start.sh config/zookeeper.properties
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/zookeeper-server-start.sh config/zookeeper1.properties
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/zookeeper-server-start.sh config/zookeeper2.properties

Edit and add all the zookeeper nodes that running in ensemble

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/server.properties
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183

Copy of the configuration file of first kafka broker so that we can run multiple brokers

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp config/server.properties config/server1.properties
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp config/server.properties config/server2.properties

Edit with 'broker.id=1', 'port=9093', 'log.dirs=/tmp/kafka-logs1'

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/server1.properties

Edit with 'broker.id=2', 'port=9094', 'log.dirs=/tmp/kafka-logs2'

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/server2.properties

Start kafka broker

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-server-start.sh config/server.properties
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-server-start.sh config/server1.properties
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-server-start.sh config/server2.properties

Create topic

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-topics.sh --create --zookeeper localhost:2181,localhost:2182,localhost:2183 --replication-factor 3 --partitions 1 --topic clusterdemo

Start the producer

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic clusterdemo

Start consumer

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-console-consumer.sh --zookeeper localhost:2181,localhost:2182,localhost:2183 --topic clusterdemo

Check fault tolerance

Get the processID (Kill one of the zookeeper node which working on port 2181)

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ lsof -i:2181 | grep LISTEN

kill

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ kill -9 27382

Still consumer can receive the messages