Apache kafka Multi Broker Cluster Single Node Zookeeper

Apache kafka Multi Broker Cluster Single Node Zookeeper

Since we are running zookeper in single machine, we will run in mutiple ports

Create multiple configurations

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp config/server.properties config/servertwo.properties
aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ cp config/server.properties config/serverthree.properties

Change 'broker.id=1' (unique id), 'port=9093', 'log.dirs=/tmp/kafka-logs-1' (Identify logs)

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/servertwo.properties

Change 'broker.id=2' (unique id), 'port=9094', 'log.dirs=/tmp/kafka-logs-2' (Identify logs)

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ nano config/serverthree.properties

No need to change zookeeper port as all 3 brokers report to the same zookeeper

Start zookeeper

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/zookeeper-server-start.sh config/zookeeper.properties

Start first broker

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-server-start.sh config/server.properties

Start second broker

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-server-start.sh config/servertwo.properties

Start third broker

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-server-start.sh config/serverthree.properties

Create new topic

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic replicationtopic

Start the producer (list of all the broker in our cluster), mention the topic

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-console-producer.sh --broker-list localhost:9092, localhost:9093, localhost:9094 --topic replicationtopic

Start the consumer

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic replicationtopic

Check fault tolerance

Find process id

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ lsof -i:9093 |grep LISTEN

kill the process

aux-137@thashitharan:~/kafka/kafka_2.12-1.1.0$ kill -15 19168

Still consumer can receive the message