Zookeeper 集群+kafka集群+kafka manager搭建
软件需求，软件包都上传到 /usr/local/src目录：jdk-8u101-linux-x64.tar.gz
kafka.2.11-0.8.22.tar.gz
zookeeper-3.4.9.tar.gz
kafka-manager-1.3.0.7.zip
* kafka-manager是通过scala打包获取一个编译完的项目，需要提前编译好,参考 https://github.com/yahoo/kafka-manager
硬件需求，四个主机：
192.168.100.100 : kafka-manager , scala
192.168.100.101 : kafka,zookeeper
192.168.100.102 : kafka,zookeeper
192.168.100.103 : kafka,zookeeper
开始设置布置环境：
192.168.100.101,192.168.100.102,192.168.100.103 环境安装包配置如下：
# cd /usr/local/src
# ll
total 310060
-rw-r--r-- 1 root root 181352138 Oct 10 18:31 jdk-8u101-linux-x64.tar.gz
-rw-r--r-- 1 root root16038031 Oct 11 14:25 kafka.2.11-0.8.22.tar.gz
-rw-r--r-- 1 root root68699247 Oct 10 18:00 kafka-manager-1.3.0.7.zip
-rw-r--r-- 1 root root28678231 Oct 10 16:10 scala-2.11.8.tgz
-rw-r--r-- 1 root root22724574 Oct 10 16:57 zookeeper-3.4.9.tar.gz
# tar -xf jdk-8u101-linux-x64.tar.gz
# tar -xf kafka.2.11-0.8.22.tar.gz
# tar -xf zookeeper-3.4.9.tar.gz
# ll
total 310072
drwxr-xr-x8 uucp143 4096 Jun 22 18:13 jdk1.8.0_101
-rw-r--r--1 root root 181352138 Oct 10 18:31 jdk-8u101-linux-x64.tar.gz
drwxr-xr-x6 root root 4096 Feb 242016 kafka
-rw-r--r--1 root root16038031 Oct 11 14:25 kafka.2.11-0.8.22.tar.gz
-rw-r--r--1 root root68699247 Oct 10 18:00 kafka-manager-1.3.0.7.zip
-rw-r--r--1 root root28678231 Oct 10 16:10 scala-2.11.8.tgz
drwxr-xr-x 10 1001 1001 4096 Aug 23 15:42 zookeeper-3.4.9
-rw-r--r--1 root root22724574 Oct 10 16:57 zookeeper-3.4.9.tar.gz
# mv kafka /usr/local/kafka
# mv zookeeper-3.4.9 /usr/local/zookeeper
# mv jdk1.8.0_101 /usr/local/java
# echo 'export PATH=$PATH:/usr/local/java/bin/:/usr/local/kafka/bin/:/usr/local/scala/bin:/usr/local/zookeeper/' /etc/profile
# java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
开始配置zookeeper集群,三个集群机器都配置如下:
## 编辑/usr/local/zookeeper/conf/zoo.cnf文件,内容如下:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
dataLogDir=/usr/local/zookeeper/log
clientPort=2181
maxClientCnxns=300
server.1=192.168.100.101:2888:3888
server.2=192.168.100.102:2888:3888
server.3=192.168.100.103:2888:3888
# mkdir -p /data/zookeeper
# mkdir -p /usr/local/zookeeper/log
#192.168.100.101 在zookeeper 中id是1 执行下面
# echo 1 > /data/zookeeper/myid
#192.168.100.102 在zookeeper 中id是2 执行下面
# echo 2 > /data/zookeeper/myid
#192.168.100.103 在zookeeper 中id是3 执行下面
# echo 3 > /data/zookeeper/myid 三台机器的防火墙放开zookeeper端口 2181,2888,3888
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 2181 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 2888 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 3888 -j ACCEPT
启动zookeeper,启动完，会自动推选其中一台是leader
# cd/usr/local/zookeeper/bin
# ./zkServer.sh start
# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
开始配置kafka集群,三个集群机器都配置如下:
编辑 /usr/local/kafka/config/server.properties
##broker标识,每个broker靠这个标识区别
## 192.168.100.101 是 1
## 192.168.100.102 是 2
## 192.168.100.102 是 3
broker.id=1
## 用来侦听连接的端口，生产者或消费者连接该端口
port=9990
## 指定broker网络地址，根据kafka所在的ip填写
host.name=192.168.100.101
#host.name=192.168.100.102
#host.name=192.168.100.103
num.network.threads=4
##每个分区的备份个数，默认为，值过大可能导致同步时延迟大
num.partitions=3
## zookeeper 集群，kafka需要zookeeper来保存meata信息
zookeeper.connect=192.168.100.101:2181,192.168.100.102:2181,192.168.100.103:2181
## 同CPU核数一样就可以
num.io.threads=4
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
##日志文件保存目录
log.dirs=/data/kafka
num.recovery.threads.per.data.dir=1
log.segment.bytes=1073741824
## 日志保留最长时间小时,以及日志策略
log.retention.hours=24
log.retention.check.interval.ms=300000
log.cleaner.enable=true
log.cleanup.policy=delete
zookeeper.connection.timeout.ms=6000
controller.message.queue.size=10
## 每个topic默认分片存储数量
default.replication.factor=2
## 允许上传topic
delete.topic.enable=true
auto.create.topics.enable=false 编辑 /usr/local/kafka/bin/kafka-server-start.sh ：
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
#export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi
# 修改上面的内容变成，KAFKA_HEAP_OPTS这个根据自己的主机进行配置，测试机器是4核8G，所以采用下面的配置：
# 这个配置成本机的IP -Djava.rmi.server.hostname=192.168.100.101
# 开启jvm 的rmi配置可以让kafka-manager提供更详细的kafka操作统计数据
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export KAFKA_HEAP_OPTS="-Xmx3G -Xms3G -Xmn1G"
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.ho
stname=192.168.100.101"
fi 启动kafka
# /usr/local/kafka/bin/kafka-server-start.sh -daemon/usr/local/kafka/config/server.properties
## 可以看到kafka进程是否启动
# ps -ef |grep java
开放iptables防火墙,8990,9990端口
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8999 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 9990 -j ACCEPT
测试kafka集群
# cd /usr/local/kafka/bin/
# ./kafka-topics.sh --zookeeper 192.168.100.101:2181,192.168.100.102:2181,192.168.100.103:2181 --create--topic test
# ./kafka-topics.sh --zookeeper 192.168.100.101:2181,192.168.100.102:2181,192.168.100.103:2181--list
test
192.168.100.100 安装kafka-manager业务
# cd /usr/local/src
# unzipkafka-manager-1.3.0.7.zip
# mv kafka-manager-1.3.0.7 /usr/local/kafka-manager
配置kafka-manager的配置文件 /usr/local/kafka-manager/conf/application.conf
##修改zookeeper集群ip
kafka-manager.zkhosts="192.168.100.101:2181,192.168.100.102:2181,192.168.100.103:2181"
启动kafka-manager， 防火墙放开9000端口
# cd/usr/local/kafka-manager/bin
# ./kafka-manager
浏览器访问 http://192.168.100.100:9000,可以看到下面界面,说明安装完成
http://s4.运维网.com/wyfs02/M02/8A/37/wKiom1gqyRiyf9jzAAA_K9pu3HE838.png-wh_500x0-wm_3-wmp_4-s_108365169.png
添加一个集群进入,填写后，其它默认，点击save,那么就可以看到标题已经添加
http://s1.运维网.com/wyfs02/M02/8A/34/wKioL1gqywWTEGnAAABs8srE9IA016.png-wh_500x0-wm_3-wmp_4-s_241303094.png
由于都是web操作，比较简单，其它的功能可以自己试试，这个不是重点，这里不多说
kafka集群压力测试
# cd /usr/local/kafka/bin/
## 消费者压力测试,单线程压500w条消息
#./kafka-consumer-perf-test.sh --zookeeper 192.168.100.101:2181,192.168.100.102:2181,192.168.100.103:2181 --messages 50000000 --topic test --threads 1
## 生产者压力测试,8个线程，压500w条消息，每条消息大小100B,每批10000条
#./kafka-producer-perf-test.sh --messages 5000000 --message-size 100--batch-size 10000 --topics test--threads 8 --broker-list 192.168.100.101:2181,192.168.100.102:2181,192.168.100.103:2181 由于搭建的环境已经在生产使用，就不截图列出
页:
[1]