【Kafka四】Kakfa伪分布式安装

fumingxia

　　在http://bit1129.iteye.com/blog/2174791一文中，实现了单Kafka服务器的安装，在Kafka中，每个Kafka服务器称为一个broker。本文简单介绍下，在单机环境下Kafka的伪分布式安装和测试验证

 

1. 安装步骤
　　Kafka伪分布式安装的思路跟Zookeeper的伪分布式安装思路完全一样，不过比Zookeeper稍微简单些(不需要创建myid文件),主要是针对每个Kafka服务器配置一个单独的server.properties，三个服务器分别使用server.properties，server.1.properties, server.2.properties

cp server.properties server.1.properties
cp server.properties server.2.properties
　　修改server.1.properties和server.2.properties,主要有三个属性需要修改

broker.id=1
port=9093
log.dirs=/tmp/kafka-logs-1
 
　　port指的是Kakfa服务器监听的端口
　　启动三个Kafka:
bin/kafka-server-start.sh server.properties
　　bin/kafka-server-start.sh server.1.properties
　　bin/kafka-server-start.sh server.2.properties

2. Kafka脚本常用配置参数

2.1 kafka-console-consumer.sh
　　--from-beginning                        If the consumer does not already have an established offset to consume from, start with the earliest message present in the log rather than the latest message. 
　　--topic <topic>                           The topic id to consume on
　　--zookeeper <urls>                    REQUIRED: The connection string for the zookeeper connection in the form host:port. Multiple URLS can be given to allow fail-over.
　　--group <gid>                            The group id to consume on. (default: console-consumer-37803)
　　在consumer端，不需要指定broke-list，而是通过zookeeper和topic找到所有的持有topic消息的broker

2.2 kafka-console-producer.sh
　　--topic <topic>                         REQUIRED: The topic id to produce  messages to.
　　--broker-list <broker-list>        REQUIRED: The broker list string in the form HOST1:PORT1,HOST2:PORT2.

 

2.3 kafka-topic.sh
　　--create                                Create a new topic.
　　--describe                              List details for the given topics.
　　--list                                  List all available topics.
　　--partitions <Integer: # of partitions> The number of partitions for the topic being created or altered (WARNING:   If partitions are increased for a  topic that has a key, the partition logic or ordering of the messages will be affected)
　　--replication-factor <Integer: replication factor> The replication factor for each partition in the topic being created
　　--zookeeper <urls>                    REQUIRED: The connection string for the zookeeper connection in the form host:port. Multiple URLS can be given to allow fail-over.
　　--topic <topic>                         The topic to be create, alter or describe. Can also accept a regular expression except for --create option

3. 伪机群测试
　　测试前，先总结有哪些测试点
　　目前想到的是，Partition有个leader的概念，leader partition是什么意思？干什么用的？

3.1 创建Topic

./kafka-topics.sh --create  --topic  topic_p10_r3 --partitions 10 --replication-factor 3  --zookeeper localhost:2181
　　创建一个Topic，10个Partition，副本数为3，也就是说，每个broker上的每个分区，在其它节点都有副本，因为每个节点都有10个节点的数据

3.2 每个broker创建的目录
　　当创建完Topic后，每个Topic都会在Kakfa的配置目录下（比如/tmp/kafka-logs，建立相应的目录和文件）
　　topic_p10_r3-0
　　topic_p10_r3-1
　　----
　　topic_p10_r3-9
　　其中每个目录下面都有两个文件： 00000000000000000000.index  00000000000000000000.log

3.3 Topic的详细信息

./kafka-topics.sh --describe --topic topic_p10_r3  --zookeeper localhost:2181
　　得到的结果如下：

Topic:topic_p10_r3PartitionCount:10ReplicationFactor:3Configs:
Topic: topic_p10_r3Partition: 0Leader: 2Replicas: 2,0,1Isr: 2,0,1
Topic: topic_p10_r3Partition: 1Leader: 0Replicas: 0,1,2Isr: 0,1,2
Topic: topic_p10_r3Partition: 2Leader: 1Replicas: 1,2,0Isr: 1,2,0
Topic: topic_p10_r3Partition: 3Leader: 2Replicas: 2,1,0Isr: 2,1,0
Topic: topic_p10_r3Partition: 4Leader: 0Replicas: 0,2,1Isr: 0,2,1
Topic: topic_p10_r3Partition: 5Leader: 1Replicas: 1,0,2Isr: 1,0,2
Topic: topic_p10_r3Partition: 6Leader: 2Replicas: 2,0,1Isr: 2,0,1
Topic: topic_p10_r3Partition: 7Leader: 0Replicas: 0,1,2Isr: 0,1,2
Topic: topic_p10_r3Partition: 8Leader: 1Replicas: 1,2,0Isr: 1,2,0
Topic: topic_p10_r3Partition: 9Leader: 2Replicas: 2,1,0Isr: 2,1,0
　　具体的含义是：
　　Here is an explanation of output. The first line gives a summary of all the partitions, each additional line gives information about one partition

• "leader" is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
  
• "replicas" is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
  
• "isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.
  

　　3.4 问题： 如果副本数为1，是否表示每个partition在集群中只有1份(也就是说每个partition只会存在于一个broker上)，那么leader自然就表示这个partition就在leader所指的broker上了？
　　建立包含10个分区，同时只有一个副本的topic

./kafka-topics.sh --create  --topic  topic_p10_r1 --partitions 10 --replication-factor 1  --zookeeper localhost:2181
　　详细信息：

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r1  --zookeeper localhost:2181
Topic:topic_p10_r1PartitionCount:10ReplicationFactor:1Configs:
Topic: topic_p10_r1Partition: 0Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 1Leader: 2Replicas: 2Isr: 2
Topic: topic_p10_r1Partition: 2Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 3Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 4Leader: 2Replicas: 2Isr: 2
Topic: topic_p10_r1Partition: 5Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 6Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 7Leader: 2Replicas: 2Isr: 2
Topic: topic_p10_r1Partition: 8Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 9Leader: 1Replicas: 1Isr: 1
　　可见理解不错，每个partition有不同的leader，Leader所在的broker同时也是Replicas所在的broker(ID号一样)
　　因此可以理解，
　　1. 每个partition副本集都有一个leader
　　2. leader指的是partition副本集中的leader，它负责读写，然后负责将数据复制到其它的broker上。
　　3.一个Topic的所有partition会比较均匀的分布到多个broker上

3.5 broker挂了，Kafka的容错机制
　　在上面已经建立了两个Topic，一个是10个分区3个副本， 一个是10个分区1个副本。此时，假如有一个broker挂了，看看这两个Topic的容错如何？
　　通过jps命令可以看到有三个Kafka进程。
　　通过ps -ef|grep server.2.properties可以找到brokerId为2的Kakfa进程，使用kill -9将其干掉。干掉的时候，console开始刷屏，异常信息一样，都是：

[2015-02-23 02:14:00,037] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
[2015-02-23 02:14:00,039] ERROR [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 4325; ClientId: ReplicaFetcherThread-0-2; ReplicaId: 1; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: [topic_p10_r3,3] -> PartitionFetchInfo(0,1048576),[topic_p10_r3,9] -> PartitionFetchInfo(0,1048576),[topic_p10_r3,6] -> PartitionFetchInfo(0,1048576),[topic_p10_r3,0] -> PartitionFetchInfo(0,1048576) (kafka.server.ReplicaFetcherThread)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
at kafka.consumer.SimpleConsumer.reconnect(SimpleConsumer.scala:57)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:79)
at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2015-02-23 02:14:00,040] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)
　　3,9，6,0正是topic_p10_r3上broker2作为leader的partition，可见Kafka要做Leader移交，看看此时的topic_p10_r3和topic_p10_r1的情况
　　topic_p10_r3（Partition切换到其它Leader上了。。。Rplicas还有3,。。。）

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r3  --zookeeper localhost:2181
Topic:topic_p10_r3PartitionCount:10ReplicationFactor:3Configs:
Topic: topic_p10_r3Partition: 0Leader: 0Replicas: 2,0,1Isr: 0,1
Topic: topic_p10_r3Partition: 1Leader: 0Replicas: 0,1,2Isr: 0,1
Topic: topic_p10_r3Partition: 2Leader: 1Replicas: 1,2,0Isr: 1,0
Topic: topic_p10_r3Partition: 3Leader: 1Replicas: 2,1,0Isr: 1,0
Topic: topic_p10_r3Partition: 4Leader: 0Replicas: 0,2,1Isr: 0,1
Topic: topic_p10_r3Partition: 5Leader: 1Replicas: 1,0,2Isr: 1,0
Topic: topic_p10_r3Partition: 6Leader: 0Replicas: 2,0,1Isr: 0,1
Topic: topic_p10_r3Partition: 7Leader: 0Replicas: 0,1,2Isr: 0,1
Topic: topic_p10_r3Partition: 8Leader: 1Replicas: 1,2,0Isr: 1,0
Topic: topic_p10_r3Partition: 9Leader: 1Replicas: 2,1,0Isr: 1,0
　　topic_p10_r1:没有切换，但是Leader是-1了。。

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r1  --zookeeper localhost:2181
Topic:topic_p10_r1PartitionCount:10ReplicationFactor:1Configs:
Topic: topic_p10_r1Partition: 0Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 1Leader: -1Replicas: 2Isr:
Topic: topic_p10_r1Partition: 2Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 3Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 4Leader: -1Replicas: 2Isr:
Topic: topic_p10_r1Partition: 5Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 6Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 7Leader: -1Replicas: 2Isr:
Topic: topic_p10_r1Partition: 8Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 9Leader: 1Replicas: 1Isr: 1
　　重启broker 2得到结果如下：（对于topic_p10_r3，leader没有变化，即每个Partition都有自己的Leader，新加入的broker只能follower；而topic_p10_r1,则会选出Leader）

[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r3  --zookeeper localhost:2181
Topic:topic_p10_r3PartitionCount:10ReplicationFactor:3Configs:
Topic: topic_p10_r3Partition: 0Leader: 0Replicas: 2,0,1Isr: 0,1,2
Topic: topic_p10_r3Partition: 1Leader: 0Replicas: 0,1,2Isr: 0,1,2
Topic: topic_p10_r3Partition: 2Leader: 1Replicas: 1,2,0Isr: 1,0,2
Topic: topic_p10_r3Partition: 3Leader: 1Replicas: 2,1,0Isr: 1,0,2
Topic: topic_p10_r3Partition: 4Leader: 0Replicas: 0,2,1Isr: 0,1,2
Topic: topic_p10_r3Partition: 5Leader: 1Replicas: 1,0,2Isr: 1,0,2
Topic: topic_p10_r3Partition: 6Leader: 0Replicas: 2,0,1Isr: 0,1,2
Topic: topic_p10_r3Partition: 7Leader: 0Replicas: 0,1,2Isr: 0,1,2
Topic: topic_p10_r3Partition: 8Leader: 1Replicas: 1,2,0Isr: 1,0,2
Topic: topic_p10_r3Partition: 9Leader: 1Replicas: 2,1,0Isr: 1,0,2
[hadoop@hadoop bin]$ ./kafka-topics.sh --describe --topic topic_p10_r1  --zookeeper localhost:2181
Topic:topic_p10_r1PartitionCount:10ReplicationFactor:1Configs:
Topic: topic_p10_r1Partition: 0Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 1Leader: 2Replicas: 2Isr: 2
Topic: topic_p10_r1Partition: 2Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 3Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 4Leader: 2Replicas: 2Isr: 2
Topic: topic_p10_r1Partition: 5Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 6Leader: 1Replicas: 1Isr: 1
Topic: topic_p10_r1Partition: 7Leader: 2Replicas: 2Isr: 2
Topic: topic_p10_r1Partition: 8Leader: 0Replicas: 0Isr: 0
Topic: topic_p10_r1Partition: 9Leader: 1Replicas: 1Isr: 1