Apache Kafka分布式消息队列安装配置
一、介绍Apache Kafka是由Apache软件基金会开发的一个开源消息系统项目，由Scala写成。Kafka最初是由LinkedIn开发，并于2011年初开源。2012年10月从Apache Incubator毕业。该项目的目标是为处理实时数据提供一个统一、高通量、低等待的平台。
二、安装环境
kafka server IP:
1
10.0.0.25 10.0.0.26
操作系统：
1
CentOS 6.5 x86_64
须用到的软件包:
1
jdk-1.7.0_65-fcs.x86_64 java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
Hosts表记录
1
2
3
cat /etc/hosts
10.0.0.25 kafka1
10.0.0.26 kafka2
三、配置kafka的两个节点(kafka1和kafka2)
配置kafka1
1、解压kafka_2.10-0.8.1.1.tgz到kafka1
1
2
3
tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/
cd /usr/local/
mv kafka_2.10-0.8.1.1/ kafka
2、配置kafka1上的/usr/local/kafka/config/server.properties
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
egrep -v '(^#|^$)' /usr/local/kafka/config/server.properties
broker.id=1
port=9000
num.network.threads=2
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/usr/local/kafka/kafka-logs
num.partitions=8
log.retention.hours=168
log.segment.bytes=536870912
log.retention.check.interval.ms=60000
log.cleaner.enable=false
zookeeper.connect=kafka1:2181,kafka2:2181
zookeeper.connection.timeout.ms=1000000
1
2
3
4
egrep -v '(^#|^$)' /usr/local/kafka/config/zookeeper.properties
dataDir=/usr/local/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
配置kafka2
1、解压kafka_2.10-0.8.1.1.tgz到kafka1
1
2
3
tar -xf kafka_2.10-0.8.1.1.tgz -C /usr/local/
cd /usr/local/
mv kafka_2.10-0.8.1.1/ kafka
2、配置kafka2上的/usr/local/kafka/config/server.properties
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
egrep -v '(^#|^$)' /usr/local/kafka/config/server.properties
broker.id=2
port=9000
num.network.threads=2
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/usr/local/kafka/kafka-logs
num.partitions=8
log.retention.hours=168
log.segment.bytes=536870912
log.retention.check.interval.ms=60000
log.cleaner.enable=false
zookeeper.connect=kafka1:2181,kafka2:2181
zookeeper.connection.timeout.ms=1000000
1
2
3
4
egrep -v '(^#|^$)' /usr/local/kafka/config/zookeeper.properties
dataDir=/usr/local/kafka/zookeeper
clientPort=2181
maxClientCnxns=0
四、编写kafka启动、关闭shell脚本
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
cat /etc/init.d/kafka
#!/bin/bash
source /etc/profile;
function Stop()
{
ps -ef |grep kafka|grep -v grep |awk '{print $2}'|xargs kill -9
}
function Start()
{
/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &> /dev/null &
sleep 1
/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &> /dev/null &
}
case $1 in
stop)
Stop
;;
start)
Start
;;
*)
echo "Usage: `basename $0` {stop|start}"
;;
esac
用法：
1
2
3
4
# 启动kafka：
/etc/init.d/kafka start
# 关闭kafka：
/etc/init.d/kafka stop
五、让kafka开机自启动
1
echo '/etc/init.d/kafka start > /dev/null ' >> /etc/rc.local
页:
[1]