Apache Kafka分布式消息队列安装配置

0987

一、介绍

Apache Kafka是由Apache软件基金会开发的一个开源消息系统项目，由Scala写成。Kafka最初是由LinkedIn开发，并于2011年初开源。2012年10月从Apache Incubator毕业。该项目的目标是为处理实时数据提供一个统一、高通量、低等待的平台。

二、安装环境
kafka server IP:

1	10.0.0.25 10.0.0.26

操作系统：

1	CentOS 6.5 x86_64

须用到的软件包:   

1	jdk-1.7.0_65-fcs.x86_64 java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

Hosts表记录

1 2 3

cat /etc/hosts 10.0.0.25 kafka1 10.0.0.26 kafka2

三、配置kafka的两个节点(kafka1和kafka2)

配置kafka1

1、解压kafka_2.10-0.8.1.1.tgz到kafka1

1 2 3

tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/ cd /usr/local/ mv kafka_2.10-0.8.1.1/ kafka

2、配置kafka1上的/usr/local/kafka/config/server.properties

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

egrep -v '(^#|^$)' /usr/local/kafka/config/server.properties broker.id=1 port=9000 num.network.threads=2    num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/usr/local/kafka/kafka-logs num.partitions=8 log.retention.hours=168 log.segment.bytes=536870912 log.retention.check.interval.ms=60000 log.cleaner.enable=false zookeeper.connect=kafka1:2181,kafka2:2181 zookeeper.connection.timeout.ms=1000000

1 2 3 4

egrep -v '(^#|^$)' /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeper clientPort=2181 maxClientCnxns=0

配置kafka2

1、解压kafka_2.10-0.8.1.1.tgz到kafka1

1 2 3

tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/ cd /usr/local/ mv kafka_2.10-0.8.1.1/ kafka

2、配置kafka2上的/usr/local/kafka/config/server.properties

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

egrep -v '(^#|^$)' /usr/local/kafka/config/server.properties broker.id=2 port=9000 num.network.threads=2    num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/usr/local/kafka/kafka-logs num.partitions=8 log.retention.hours=168 log.segment.bytes=536870912 log.retention.check.interval.ms=60000 log.cleaner.enable=false zookeeper.connect=kafka1:2181,kafka2:2181 zookeeper.connection.timeout.ms=1000000

1 2 3 4

egrep -v '(^#|^$)' /usr/local/kafka/config/zookeeper.properties dataDir=/usr/local/kafka/zookeeper clientPort=2181 maxClientCnxns=0

四、编写kafka启动、关闭shell脚本

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

cat /etc/init.d/kafka #!/bin/bash source /etc/profile; function Stop() { ps -ef |grep kafka|grep -v grep |awk '{print $2}'|xargs kill -9 } function Start() { /bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &> /dev/null & sleep 1 /bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &> /dev/null & } case $1 in         stop)             Stop             ;;         start)             Start             ;;         *)             echo "Usage: `basename $0` {stop|start}"             ;; esac

用法：

1 2 3 4

# 启动kafka： /etc/init.d/kafka start # 关闭kafka： /etc/init.d/kafka stop

五、让kafka开机自启动

1	echo '/etc/init.d/kafka start > /dev/null ' >> /etc/rc.local