Kafka安装

zxcvb12

　　一、下载kafka：
　　http://kafka.apache.org/downloads
　　二、解压
　　tar -zxvf kafka_2.10-0.10.0.1.tgz
　　三、kafka需要用到zookeeper，可以是单节点，也可以是zk集群。
　　（1）、单节点zk
　　kafka本身自带了一个测试zk，可以使用kafka自带的zk节点来测试。
　　1、启动单节点zookeeper



bin/zookeeper-server-start.sh config/zookeeper.properties

　　2、启动kafka 服务：



bin/kafka-server-start.sh config/server.properties

　　3、创建一个topic



bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

　　4、创建一个produce，生产者角色，产生数据，并发送给kafka



bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
　　5、创建一个 consumer，消费者角色，消费数据，接收由produce产生，kafka传递过来的数据。



bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

　　在produce控制台输入一些字符，就可以在消费者控制台看到数据了。



[hadoop@master2 kafka_2.10-0.10.0.1]$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
java
This is a message
This is another message

　　（2）、zk集群模式：
　　1、编辑vi config/server.properties 文件，将配置文件中zookeeper的地址改成zk集群节点和kafka数据存放路径



#zookeeper.connect=localhost:2181
zookeeper.connect=node1:2181,node2:2181,node3:2181
# kafka数据存放路径
# A comma seperated list of directories under which to store log files
log.dirs=/data/kafka_2.10-0.10.0.1/message-folder

　　2、启动zk集群
　　3、使用修改后的server.properties文件启动kafka



bin/kafka-server-start.sh config/server.properties

　　启动日志：



[hadoop@master2 kafka_2.10-0.10.0.1]$ bin/kafka-server-start.sh config/server.properties
[2016-10-09 01:21:38,298] INFO KafkaConfig values:
request.timeout.ms = 30000
log.roll.hours = 168
inter.broker.protocol.version = 0.10.0-IV1
log.preallocate = false
security.inter.broker.protocol = PLAINTEXT
.......
(kafka.server.KafkaConfig)
[2016-10-09 01:21:38,373] INFO starting (kafka.server.KafkaServer)
[2016-10-09 01:21:38,383] INFO Connecting to zookeeper on node1:2181,node2:2181,node3:2181 (kafka.server.KafkaServer)
[2016-10-09 01:21:38,414] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,414] INFO Client environment:host.name=master2 (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,415] INFO Client environment:java.version=1.7.0_79 (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,428] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,428] INFO Client environment:java.home=/data/jdk1.7.0_79/jre (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,429] INFO Client environment:java.class.path=:/data/kafka_2.10-0.10.0.1/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/argparse4j-0.5.0.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/connect-api-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/connect-file-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/connect-json-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/connect-runtime-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/guava-18.0.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/hk2-api-2.4.0-b34.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/hk2-locator-2.4.0-b34.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/hk2-utils-2.4.0-b34.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jackson-annotations-2.6.0.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jackson-core-2.6.3.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jackson-databind-2.6.3.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/javassist-3.18.2-GA.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/javax.annotation-api-1.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/javax.inject-1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/javax.inject-2.4.0-b34.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/javax.ws.rs-api-2.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jersey-client-2.22.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jersey-common-2.22.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jersey-container-servlet-2.22.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jersey-guava-2.22.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jersey-media-jaxb-2.22.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jersey-server-2.22.2.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-http-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-io-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-security-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-server-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jetty-util-9.2.15.v20160210.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/jopt-simple-4.9.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka_2.10-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka_2.10-0.10.0.1-sources.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka_2.10-0.10.0.1-test-sources.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka-clients-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka-log4j-appender-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka-streams-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka-streams-examples-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/kafka-tools-0.10.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/log4j-1.2.17.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/lz4-1.3.0.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/metrics-core-2.2.0.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/osgi-resource-locator-1.0.1.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/reflections-0.9.10.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/rocksdbjni-4.8.0.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/scala-library-2.10.6.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/slf4j-api-1.7.21.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/slf4j-log4j12-1.7.21.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/snappy-java-1.1.2.6.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/validation-api-1.1.0.Final.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/zkclient-0.8.jar:/data/kafka_2.10-0.10.0.1/bin/../libs/zookeeper-3.4.6.jar (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,430] INFO Client environment:java.library.path=/usr/java/packages/lib/i386:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,430] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,430] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,430] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,431] INFO Client environment:os.arch=i386 (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,431] INFO Client environment:os.version=2.6.18-92.el5 (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,431] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,431] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,431] INFO Client environment:user.dir=/data/kafka_2.10-0.10.0.1 (org.apache.zookeeper.ZooKeeper)
[2016-10-09 01:21:38,433] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
...........
[2016-10-09 01:21:39,870] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser)
[2016-10-09 01:21:39,872] INFO [Kafka Server 0], started (kafka.server.KafkaServer)

　　4、创建一个topic



bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 1 --partitions 1 --topic test

　　如果该topic已经存在，就报错：



[2016-10-09 01:23:35,106] ERROR kafka.common.TopicExistsException: Topic "test" already exists.
at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:420)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:404)
at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:110)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
(kafka.admin.TopicCommand$)

　　5、查看已经创建的topic



[hadoop@master2 kafka_2.10-0.10.0.1]$ bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181
test

　　6、创建一个数据生产者



bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

　　7、创建一个数据消费者



bin/kafka-console-consumer.sh --zookeeper node1:2181,node2:2181,node3:2181 --topic test --from-beginning

　　测试：
　　在数据生产者控制台输入数据
　　在数据消费者控制台可以看到相应的数据：



[hadoop@master2 kafka_2.10-0.10.0.1]$ bin/kafka-console-consumer.sh --zookeeper node1:2181,node2:2181,node3:2181 --topic test --from-beginning
java
This is a message
This is another message

　　四、安装kafka集群
　　我使用两台机安装了两个kafka节点。
　　1、把kafka复制到其他机器上去，
　　2、修改config/server.properties文件，分别把broker.id改为其他数字，一定要是正数，不能跟其他节点相同



broker.id=2

　　3、分别启动kafka



bin/kafka-server-start.sh config/server.properties
　　4、如果server.properties文件里配置（即log.dirs配置项）的kafka数据存放目录下，meta数据已经存在，需要清空该文件夹。否则可能会报以下错误。
　　或者修改kafka数据存放目录下meta.properties文件中broker.id配置项，使之跟server.properties中的broker.id一致。



[2016-10-12 00:09:10,898] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured broker.id 1 doesn't match stored broker.id 0 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:648)
at kafka.server.KafkaServer.startup(KafkaServer.scala:187)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-10-12 00:09:10,900] INFO shutting down (kafka.server.KafkaServer)
[2016-10-12 00:09:10,914] INFO Shutting down. (kafka.log.LogManager)
[2016-10-12 00:09:11,113] INFO Shutdown complete. (kafka.log.LogManager)
[2016-10-12 00:09:11,115] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2016-10-12 00:09:11,136] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2016-10-12 00:09:11,136] INFO Session: 0x257b7b394f70000 closed (org.apache.zookeeper.ZooKeeper)
[2016-10-12 00:09:11,140] INFO shut down completed (kafka.server.KafkaServer)
[2016-10-12 00:09:11,142] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.InconsistentBrokerIdException: Configured broker.id 1 doesn't match stored broker.id 0 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:648)
at kafka.server.KafkaServer.startup(KafkaServer.scala:187)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)

　　5、在其中一台机上创建一个topic，



bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 2 --partitions 2 --topic test-3

　　6、查看topic，已经创建成功，



[hadoop@master1 kafka_2.10-0.10.0.1]$ bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181
test-3

　　查看数据存放目录：两台机器上都有了：



[hadoop@master2 message-folder]$ ll
total 24
-rw-rw-r-- 1 hadoop hadoop    4 Oct 12 00:51 cleaner-offset-checkpoint
-rw-rw-r-- 1 hadoop hadoop   54 Oct  9 20:55 meta.properties
-rw-rw-r-- 1 hadoop hadoop   26 Oct 12 00:52 recovery-point-offset-checkpoint
-rw-rw-r-- 1 hadoop hadoop   26 Oct 12 00:52 replication-offset-checkpoint
drwxrwxr-x 2 hadoop hadoop 4096 Oct 12 00:52 test-3-0
drwxrwxr-x 2 hadoop hadoop 4096 Oct 12 00:52 test-3-1

　　kafka集群安装成功。
　　五、server.properties常用配置项：



broker.id=0 # kafka节点id，必须是正数，不能相同
num.network.threads=2 # kafka处理消息的线程数
num.io.threads=8 #kafka IO线程数
# 等待IO线程处理的请求队列最大数
queued.max.requests = 500
# socket发送数据的缓冲区大小
socket.send.buffer.bytes=1048576  
# socket接收数据的缓冲区大小
socket.receive.buffer.bytes=1048576
# socket请求的最大字节数
socket.request.max.bytes=104857600
# kafka数据存放目录，多个目录使用逗号分隔
log.dirs=/data/kafka_2.10-0.10.0.1/message-folder
# 分区数量
num.partitions=2
# 数据保存时间，单位：小时，默认是7天
log.retention.hours=168
# 日志segment文件的大小的上限，-1表示不限制。
log.segment.bytes=536870912
# 日志片段文件的检查周期，查看它们是否达到了删除策略的设置（log.retention.hours或log.retention.bytes
log.retention.check.interval.ms=60000
# 是否开启压缩
log.cleaner.enable=false
# 对于压缩的日志保留的最长时间
log.cleaner.delete.retention.ms = 1 day
#zookeeper连接地址，多个用逗号分隔
zookeeper.connect=localhost:2181
# zookeeper连接超时时间
zookeeper.connection.timeout.ms=1000000

　　六、常用命令：
　　（1）、kafka-topics.sh 脚本命令
　　1、脚本参数



--alter           修改topic分区配置，比如分区数量，replica assignment等。
--config          配置项，
--create          创建一个topic  
--delete          删除一个topic
--delete-config   删除一个topic配置项
--describe        列出topic详细信息
--disable-rack-aware           Disable rack aware replica assignment
--help                            打印帮助选项
--if-exists                       在alter、删除一个topic时，仅在topic存在时执行
--if-not-exists                   创建一个topic时，在topic不存在时执行
--list                            列出所有可用topic
--partitions                      设置分区数
--replica-assignment              A list of manual partition-to-broker
--topic                           设置topic名
--topics-with-overrides           if set when describing topics, only how topics that have overridden configs
--unavailable-partitions          在列出topic信息（即describe）时，列出不用的分区
--under-replicated-partitions     if set when describing topics, only show under replicated partitions  
--zookeeper                       zookeeper连接地址，格式host:port,host:port

　　示例：　
　　1、创建一个topic
　　创建一个名为test-1，partition备份数为1，分区数为1的topic。



bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 1 --partitions 1 --topic test-1

　　注意，partition备份数不可以超过kafka集群的数量，分区数可以。
　　2、查看topic列表



bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181
　　3、删除一个topic，



bin/kafka-topics.sh --delete --zookeeper node1:2181,node2:2181,node3:2181 --topic test-3

　　再查看topic列表，其实并没有立刻删除。。



bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181

　　控制台显示：Topic test-3 is marked for deletion.
　　解决办法：
　　A。手动删除方法：
　　先删除每个broker节点的topic数据，目录在server.properties文件的log.dirs配置项，以要删除的topic 名字开头的文件夹。
　　再删除zookeeper的数据：
　　rmr /brokers/topics/{topic_name}
  rmr /admin/delete_topics/{topic_name}
　　rmr /config/topics/{topic_name}
　　B、kafka自动立刻删除：
　　需要设置在启动broker时候开启删除topic的开关，即在server.properties中添加：



delete.topic.enable=true
　　参考：http://kafka.apache.org/quickstart