Zookeeper和zkpython

wumai

　　简介：
　　Zookeeper (http://zookeeper.apache.org/)分布式服务框架是 Apache Hadoop 的一个子项目，它主要是用来解决分布式应用中经常遇到的一些数据管理问题，如：统一命名服务、状态同步服务、集群管理、分布式应用配置项的管理等。
　　具体简介可以参照这篇文章。
　　zkpython的安装：
　　python中有一个zkpython的包，是基于zookeeper的c-client开发的，所以安装的时候需要先安装zookeeper的c客户端。安装步骤如下：

1 2 3 4 5 6 7 8 9 10 11 12 13

# 首先下载zookeeper wget http://labs.renren.com/apache-mirror//zookeeper/zookeeper-3.3.3/zookeeper-3.3.3.tar.gz tar xzvf zookeeper-3.3.3.tar.gz cd zookeeper-3.3.3/src/c/ ./configure make make install # 然后下载zkpython wget http://pypi.python.org/packages/source/z/zkpython/zkpython-0.4.tar.gz#md5=3de220615aaddf57f1462b78d32477f9 tar xzvf zkpython-0.4.tar.gz cd zkpython-0.4 python setup.py install

　　这样就完成了zkpython的安装。
　　一个简单的demo：
　　之后让我们来写一个简单的demo吧。（demo中用到的zkclient.py:https://github.com/piglei/zkpython_example/blob/master/zkclient.py）

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

# coding: utf-8 import logging from os.path import basename, join from zkclient import ZKClient, zookeeper, watchmethod logging.basicConfig(     level = logging.DEBUG,     format = "[%(asctime)s] %(levelname)-8s %(message)s" ) log = logging class GJZookeeper(object):     ZK_HOST = "localhost:2181"     ROOT = "/app"     WORKERS_PATH = join(ROOT, "workers")     MASTERS_NUM = 1     TIMEOUT = 10000     def __init__(self, verbose = True):         self.VERBOSE = verbose         self.masters = []         self.is_master = False         self.path = None         self.zk = ZKClient(self.ZK_HOST, timeout = self.TIMEOUT)         self.say("login ok!")         # init         self.__init_zk()         # register         self.register()     def __init_zk(self):         """         create the zookeeper node if not exist         """         nodes = (self.ROOT, self.WORKERS_PATH)         for node in nodes:             if not self.zk.exists(node):                 try:                     self.zk.create(node, "")                 except:                     pass     @property     def is_slave(self):         return not self.is_master     def register(self):         """         register a node for this worker         """         self.path = self.zk.create(self.WORKERS_PATH + "/worker", "1", flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE)         self.path = basename(self.path)         self.say("register ok! I'm %s" % self.path)         # check who is the master         self.get_master()     def get_master(self):         """         get children, and check who is the smallest child         """         @watchmethod         def watcher(event):             self.say("child changed, try to get master again.")             self.get_master()         children = self.zk.get_children(self.WORKERS_PATH, watcher)         children.sort()         self.say("%s's children: %s" % (self.WORKERS_PATH, children))         # check if I'm master         self.masters = children[:self.MASTERS_NUM]         if self.path in self.masters:             self.is_master = True             self.say("I've become master!")         else:             self.say("%s is masters, I'm slave" % self.masters)     def say(self, msg):         """         print messages to screen         """         if self.VERBOSE:             if self.path:                 log.info("[ %s(%s) ] %s" % (self.path, "master" if self.is_master else "slave", msg))             else:                 log.info(msg) def main():     gj_zookeeper = GJZookeeper() if __name__ == "__main__":     main()     import time     time.sleep(1000)

　　这个简单的demo所做的事情，就是通过在zookeeper的/app/workers节点下建立临时的子节点( flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE )，每次create完成之后检查自己是不是在最小的MASTERS_NUM(例子中为1，即单master)里。如果是的话，作为master运行，否则的话，作为slave运行。
　　这样的话，当我们的master挂掉以后，与zookeeper之间的连接也会中断，过了指定的TIMEOUT以后，master之前在worker下的子节点就会被删除，于是slave节点之前设置的watcher会被触发，再次检查自己是否为master，如果是的话则完成切换。
　　demo运行结果：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

# 第一个实例 Connected in 20 ms, handle is 0 [2011-09-09 12:40:43,702] INFO     login ok! Node /app/workers/worker created in 4 ms [2011-09-09 12:40:43,708] INFO     [ worker0000000022(slave) ] register ok! I'm worker0000000022 [2011-09-09 12:40:43,709] INFO     [ worker0000000022(slave) ] /app/workers's children: ['worker0000000022'] [2011-09-09 12:40:43,709] INFO     [ worker0000000022(master) ] I've become master! # 这时再起第二个实例 Connected in 64 ms, handle is 0 [2011-09-09 12:43:08,334] INFO     login ok! Node /app/workers/worker created in 11 ms [2011-09-09 12:43:08,346] INFO     [ worker0000000023(slave) ] register ok! I'm worker0000000023 [2011-09-09 12:43:08,347] INFO     [ worker0000000023(slave) ] /app/workers's children: ['worker0000000022', 'worker0000000023'] [2011-09-09 12:43:08,347] INFO     [ worker0000000023(slave) ] ['worker0000000022'] is masters, I'm slave # 杀掉master，第二个实例发生的变化 [2011-09-09 12:44:06,016] INFO     [ worker0000000023(slave) ] child changed, try to get master again. [2011-09-09 12:44:06,017] INFO     [ worker0000000023(slave) ] /app/workers's children: ['worker0000000023'] [2011-09-09 12:44:06,017] INFO     [ worker0000000023(master) ] I've become master!

　　
　　demo代码地址： https://github.com/piglei/zkpython_example