kafka迁移与扩容
参考官网site:http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
说明:
当我们对kafka集群扩容时，需要满足2点要求:
[*] 将指定topic迁移到集群内新增的node上。
[*] 将topic的指定partition迁移到新增的node上。
1. 迁移topic到新增的node上
假如现在一个kafka集群运行三个broker，broker.id依次为101,102,103,后来由于业务数据突然暴增，需要新增三个broker,broker.id依次为104,105,106.目的是要把push-token-topic迁移到新增node上。
1、脚本migration-push-token-topic.json文件内容如下:
Java代码http://liyonghui160com.iteye.com/images/icon_star.png
[*] {
[*]
[*] "topics":
[*]
[*] [
[*]
[*] {
[*]
[*] "topic": "push-token-topic"
[*]
[*] }
[*]
[*] ],
[*]
[*] "version":1
[*]
[*] }
2、执行脚本如下所示:
Java代码http://liyonghui160com.iteye.com/images/icon_star.png
[*] root@localhost:$./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183 --topics-to-move-json-filemigration-push-token-topic.json--broker-list"104,105,106"--generate
生成分配partitions的json脚本 备份恢复使用：
Current partition replica assignment
{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":},{"topic":"cluster-switch-topic","partition":5,"replicas":},{"topic":"cluster-switch-topic","partition":3,"replicas":},{"topic":"cluster-switch-topic","partition":4,"replicas":},{"topic":"cluster-switch-topic","partition":9,"replicas":},{"topic":"cluster-switch-topic","partition":1,"replicas":},{"topic":"cluster-switch-topic","partition":11,"replicas":},{"topic":"cluster-switch-topic","partition":7,"replicas":},{"topic":"cluster-switch-topic","partition":2,"replicas":},{"topic":"cluster-switch-topic","partition":0,"replicas":},{"topic":"cluster-switch-topic","partition":6,"replicas":},{"topic":"cluster-switch-topic","partition":8,"replicas":}]}
重新分配parttions的json脚本如下：
migration-topic-cluster-switch-topic.json
{"version":1,"partitions":[{"topic":"cluster-switch-topic","partition":10,"replicas":},{"topic":"cluster-switch-topic","partition":5,"replicas":},{"topic":"cluster-switch-topic","partition":4,"replicas":},{"topic":"cluster-switch-topic","partition":3,"replicas":},{"topic":"cluster-switch-topic","partition":9,"replicas":},{"topic":"cluster-switch-topic","partition":1,"replicas":},{"topic":"cluster-switch-topic","partition":11,"replicas":},{"topic":"cluster-switch-topic","partition":7,"replicas":},{"topic":"cluster-switch-topic","partition":2,"replicas":},{"topic":"cluster-switch-topic","partition":0,"replicas":},{"topic":"cluster-switch-topic","partition":6,"replicas":},{"topic":"cluster-switch-topic","partition":8,"replicas":}]}
3、执行：
Java代码http://liyonghui160com.iteye.com/images/icon_star.png
[*] root@localhost:$ bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183 --reassignment-json-file migration-topic-cluster-switch-topic.json --execute
执行后会生成一个json格式文件expand-cluster-reassignment.json
4、查询执行状态：
Java代码http://liyonghui160com.iteye.com/images/icon_star.png
[*] bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183 --reassignment-json-file expand-cluster-reassignment.json --verify
正常执行后会返回当前数据迁移的不用partion的，信息状态类似下面
Java代码http://liyonghui160com.iteye.com/images/icon_star.png
[*] Reassignment of partition completed successfully //移动成功
[*] Reassignment of partition is in progress //这行代表数据在移动中
[*] Reassignment of partition is in progress
[*] Reassignment of partition completed successfully
[*] Reassignment of partition completed successfully
这样做不会影响原来集群上的topic业务
2.topic修改(replicats-factor)副本个数
假如初始时push-token-topic为一个副本，为了提高可用性，需要改为2副本模式。
脚本replicas-update-push-token-topic.json文件内容如下:
{
"partitions":
[
{
"topic": "log.mobile_nginx",
"partition": 0,
"replicas":
},
{
"topic": "log.mobile_nginx",
"partition": 1,
"replicas":
}
],
"version":1
}
2、执行：
Java代码http://liyonghui160com.iteye.com/images/icon_star.png
[*] root@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2183 --reassignment-json-filereplicas-update-push-token-topic.json--execute
执行后会列出当前的partition和修改后的patition
3、verify
Java代码http://liyonghui160com.iteye.com/images/icon_star.png
[*] bin/kafka-reassign-partitions.sh --zookeeper 192.168.2.225:2181 --reassignment-json-file replicas-update-push-token-topic.json --verify
如下：
Status of partition reassignment:
Reassignment of partition completed successfully
Reassignment of partition completed successfully
3.自定义分区和迁移
1、The first step is to hand craft the custom reassignment plan in a json file-
> cat custom-reassignment.json
{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":},{"topic":"foo2","partition":1,"replicas":}]}
2、Then, use the json file with the --execute option to start the reassignment process-
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute
Current partition replica assignment
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":},
{"topic":"foo2","partition":1,"replicas":}]
}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":},
{"topic":"foo2","partition":1,"replicas":}]
}
3、The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --verify
Status of partition reassignment:
Reassignment of partition completed successfully
Reassignment of partition completed successfully
4.topic的分区扩容用法
a.先扩容分区数量，脚本如下:
例如:push-token-topic初始分区数量为12，目前到增加到15个
root@localhost:$ ./bin/kafka-topics.sh --zookeeper 192.168.2.225:2183 --alter --partitions 15 --topic push-token-topic
b.设置topic分区副本
root@localhost:$ ./bin/kafka-reassign-partitions.sh --zookeeper192.168.2.225:2183
--reassignment-json-file partitions-extension-push-token-topic.json--execute
脚本partitions-extension-push-token-topic.json文件内容如下:
{
"partitions":
[
{
"topic": "push-token-topic",
"partition": 12,
"replicas":
},
{
"topic": "push-token-topic",
"partition": 13,
"replicas":
},
{
"topic": "push-token-topic",
"partition": 14,
"replicas":
}
],
"version":1
}
页:
[1]