redis 3.0 cluster 集群 学习之路篇 [2]
周氏一族，整理技术文档，给下一代留点教程......Redis 3.0 cluster 基础篇，请看下面链接
http://zhoushouby.blog.51cto.com/9150272/1560346
本篇，主要是讲述，如果安装 Redis 3.0 cluster，安装步骤跟redis主备大同小异，只是需要注意两个方面
1、开启 cluster enable ，以及设置相关clustertimeout
2、Redis cluster 是用自带的ruby工具(redis-trib.rb)构建集群[在src目录里面]
准备资料：
redis-3.0.0-beta8.tar.gz 官网下载地址 http://redis.io/download/
tcl8.6.1-src.tar.gz 官网下载地址 http://sourceforge.jp/projects/sfnet_tcl/releases/
rubygems-2.4.2.zip 官网下载地址 http://rubygems.org/pages/download/
redis-3.0.0.gem 官网下载地址 http://rubygems.org/gems/redis/versions/3.1.0
系统采用的是 centos 6.364bit
# file /sbin/init
/sbin/init: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
实验环境模拟：
192.168.1.222 三个实例
192.168.1.223 三个实例
注意，由于官网，必须至少6台服务器，所以鄙人笔记本过卡，直接用两台vm来虚拟，我用不同端口号来区分不同的redis实例即可。因为启动redis时只需要指定“该redis实例的配置文件”即可。在现实当中，如果有六台物理机，那样会更加简单部署，不会像我这么复杂，不过也不复杂，就是多copy一下配置文件而已。
实验开始：
1、安装一下开发工具
yum -y install gcc make 2、安装redis依赖包tcl，这个不懂的请看redis 主备 那篇文章
http://zhoushouby.blog.51cto.com/9150272/1547084
tar xzvf tcl8.6.1-src.tar.gz
cd tcl8.6.1/
cd unix &&
./configure --prefix=/usr \
--without-tzdata \
--mandir=/usr/share/man \
$([ $(uname -m) = x86_64 ] && echo --enable-64bit) &&
make &&
sed -e "s@^\(TCL_SRC_DIR='\).*@\1/usr/include'@" \
-e "/TCL_B/s@='\(-L\)\?.*unix@='\1/usr/lib@" \
-i tclConfig.sh
# 测试make是否成功，最好不要，太烦了，要等N久，半个小时有多
make test
# root用户登录，执行下面命令
make install &&
make install-private-headers &&
ln -v -sf tclsh8.6 /usr/bin/tclsh &&
chmod -v 755 /usr/lib/libtcl8.6.so
不要以为3.0就不用这个tcl了，他扮演着很重要的角色，如果没有tcl，将会报错如下:
# make test
cd src && make test
make: Entering directory `/root/redis-3.0.0-beta8/src'
You need tcl 8.5 or newer in order to run the Redis test
make: *** Error 1
make: Leaving directory `/root/redis-3.0.0-beta8/src'
make: *** Error 2
#
3、需要先配置下面的内核参数，否则Redis脚本在重启或停止redis时，将会报错，并且不能自动在停止服务前同步数据到磁盘上
echo 1 > /proc/sys/vm/overcommit_memory
echo vm.overcommit_memory=1 >> /etc/sysctl.conf
sysctl vm.overcommit_memory=1
4、开始安装redis 3.0 cluster
cd /root/
tar xzvf redis-3.0.0-beta8.tar.gz
cd redis-3.0.0-beta8
make
make test
make install
vi /etc/redis.conf
5、编辑配置文件，修改下面几个参数
port 6379
appendonly yes
appendfilename "appendonly-6379.aof"
cluster-enabled yes
cluster-config-file /opt/nodes-6379.conf
cluster-node-timeout 5000
为什么呢？
因为，我们要在一台vm里面，虚拟三个实例
port 端口不一样，这个是肯定要的吧，不然服务都在同一个端口运行，就只能说是一个实例而已
appendonly 这个是说在后台运行，不要我们一关闭ssh，redis就挂了
appendfilename 这个默认就是appendonly.aof，如果我们不在后面加一个后缀区分，那么所有服务，他们的快照都会存放到在同一个文件 appendonly.aof，那边就会数据紊乱，A，B,C,三者数据都一样，所以，我们要区分出来，让他们一个实例，一个文件。这里还有一个主意的地方，那就是 它，不支持路径指定，你不能在 appendonly.aof 加一个 例如/opt/appendonly.aof，否则会报错，启动不了，查看日志，报错如下
# cat redis-6379.log
*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 177
>>> 'dbfilename /opt/dump-6379.rdb'
dbfilename can't be a path, just a filename
cluster-enabled 这个就不用多说了
cluster-config-file 这个是配置集群的时候，会自动生成的，同样道理，不能默认
cluster-node-timeout失效时间，设置低一些，因为我是同一台物理机的vm
6、拷贝三个实例配置文件到opt目录下
cp redis.conf /opt/redis-6379.conf 实例A的配置文件
cp redis.conf /opt/redis-6380.conf 实例B的配置文件
cp redis.conf /opt/redis-6381.conf 实例C的配置文件
7、编辑三个实例配置文件，把端口，对应改成非重叠的，为了更好的区分，我就按照
192.168.1.222 “实例A 6379” “实例B6380” “实例C6381”
192.168.1.223 “实例D 6382” “实例E6383” “实例F6384”
port appendfilename cluster-config-file 注意根据不同实例，修改成不同的值
例如，实例C的配置文件主要是
port 6381
appendfilename "appendonly-6381.aof"
cluster-config-file /opt/nodes-6381.conf
再例如，实例E的配置文件主要变化是
port 6383
appendfilename "appendonly-6383.aof"
cluster-config-file /opt/nodes-6383.conf
废话就不多说了，越说越糊涂，自己慢慢琢磨一下
8、启动redis服务
redis-server /opt/redis-6379.conf > /opt/redis-6379.log 2>&1 &
redis-server /opt/redis-6380.conf > /opt/redis-6380.log 2>&1 &
redis-server /opt/redis-6381.conf > /opt/redis-6381.log 2>&1 &
注意，为了方便查看错误信息，我把它日志，重定向到 对应的实例名字，存放在opt下面
9、一切如果正常的话，你随便找一个redis-6379.logredis-6380.log redis-6379.log
例如，我找一个 6380的日志
# cat redis-6380.log
6115:M 31 Aug 12:34:28.904 * Increased maximum number of open files to 10032 (it was originally set to 1024).
6115:M 31 Aug 12:34:28.906 * No cluster configuration found, I'm 92999f9840418a848f7b10c5bca0119e3b515fa4
_._
_.-``__ ''-._
_.-`` `.`_.''-._ Redis 2.9.57 (00000000/0) 64 bit
.-`` .-```.```\/ _.,_ ''-._
( ' , .-`| `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6380
| `-._ `._ / _.-' | PID: 6115
`-._ `-._`-./_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
6115:M 31 Aug 12:34:28.924 # Server started, Redis version 2.9.57
6115:M 31 Aug 12:34:28.924 * The server is now ready to accept connections on port 6380
10、说明一切准备就绪，都能正常运作，就差集群了，为了更加放心，看一下是否三个实例都在正常运行
# ps aux |grep redis
root 61090.60.9 1374089740 pts/1 Sl 12:33 0:52 redis-server *:6379
root 61150.60.9 1374089912 pts/2 Sl 12:34 0:51 redis-server *:6380
root 61230.60.7 1374087876 pts/2 Sl 12:35 0:50 redis-server *:6381
root 62350.00.0 103240 840 pts/2 S+ 14:47 0:00 grep redis
#
11、以上，所有步骤，在192.168.1.223，同样的做法，注意变化的就是redis 端口的改变，以及对于配置文件的变化。
12、那么，开始集群吧，集群之前，先表示怀疑一下[请注意，只需要在其中一台服务器操作下面内容]
# cd redis-3.0.0-beta8
# cd src/
# ./redis-trib.rb create --replicas 1 192.168.1.222:6379 192.168.1.222:6780 192.168.1.222:6381 192.168.1.223:6382 192.168.1.223:6383 192.168.1.223:6384
/usr/bin/env: ruby: No such file or directory
#
你会发现，他马上报错，报错信息如下：/usr/bin/env: ruby: No such file or directory
就是说，没有ruby，
因为
redis-trib位于Redis源码的src文件夹中， 它是一个Ruby程序， 这个程序通过向实例发送特殊命令来完成创建新集 群， 检查集群， 或者对集群进行重新分片（reshared）等工作。
13、既然没有ruby，那我们就yum安装一个吧
yum -y install ruby ruby-rdoc
14、好，装好了，继续来集群
# ./redis-trib.rb create --replicas 1 192.168.1.222:6379 192.168.1.222:6780
192.168.1.222:6381 192.168.1.223:6382 192.168.1.223:6383 192.168.1.223:6384
./redis-trib.rb:24:in `require': no such file to load -- rubygems (LoadError)
from ./redis-trib.rb:24
#
你会发现，又有问题了，这是什么问题啊，不过看了一下，后面提示rubygems (LoadError) ，嘿嘿，应该就是gem没有上去，那么装一个咯
15、gem下载地址，上面有 http://rubygems.org/pages/download/
官网上面有很多种哦，有TGZ ZIP GEM GIT, 我这里下载的是zip的包
16、安装rubygems
unzip rubygems-2.4.2.zip
cd rubygems-2.4.2
ruby setup.rb
刷刷刷一大堆东西出来，不过，细心的朋友，你会发现，下面有一处非常重要的 代码
# ruby setup.rb
RubyGems 2.4.2 installed
Installing ri documentation for rubygems-2.4.2
/usr/lib/ruby/1.8/rdoc/rdoc.rb:280: warning: conflicting chdir during another chdir block
/usr/lib/ruby/1.8/rdoc/rdoc.rb:287: warning: conflicting chdir during another chdir block
=== 2.4.2 / 2014-10-01
This> Bug fixes:
* RubyGems now correctly matches wildcard no_proxy hosts.Issue #997 by
voelzemo.
* Added support for missing git_source method in the gem dependencies API.
* Fixed handling of git gems with an> * Lockfiles will no longer be truncated upon resolution errors.
* Fixed messaging for `gem owner -a`.Issue #1004 by Aaron Patterson, Ryan
Davis.
* Removed meaningless ensure.Pull request #1003 by gogotanaka.
* Improved wording of --source option help.Pull request #989 by Jason Clark.
* Empty build_info files are now ignored.Issue #903 by Adan Alvarado.
* Gem::Installer ignores dependency checks when installing development
dependencies.Issue #994 by Jens Willie.
* `gem update` now continues after dependency errors.Issue #993 by aaronchi.
* RubyGems no longer warns about semantic version dependencies for the 0.x
range.Issue #987 by Jeff Felchner, pull request #1006 by Hsing-Hui Hsu.
* Added minimal lock to allow multithread installation of gems.Issue #982
and pull request #1005 by Yorick Peterse
* RubyGems now considers prerelease dependencies as it did in earlier versions
when --prerelease is given.Issue #990 by Jeremy Tryba.
* Updated capitalization in README.Issue #1010 by Ben Bodenmiller.
* Fixed activating gems from a Gemfile for default gems.Issue #991 by khoan.
* Fixed windows stub script generation for Cygwin.Issue #1000 by Brett
DiFrischia.
* Allow gem bindir and ruby.exe to live in separate diretories.Pull request
#942 by Ian Flynn.
* Fixed handling of gemspec in gem dependencies files to match Bundler
behavior.Issue #1020 by Michal Papis.
* Fixed `gem update` when updating to prereleases.Issue #1028 by Santiago
Pastorino.
* RubyGems now fails immediately when a git reference cannot be found instead
of spewing git errors.Issue #1031 by Michal Papis
=== 2.4.1 / 2014-07-17
Bug fixes:
* RubyGems can now be updated on Ruby implementations that do not support
vendordir in RbConfig::CONFIG.Issue #974 by net1957.
=== 2.4.0 / 2014-07-16
Minor enhancements:
* The contents command now supports a --show-install-dir option that shows
only the directory the gem is installed in.Feature request #966 by Akinori
MUSHA.
* Added a --build-root option to the install command for packagers.Pull
request #965 by Marcus R眉ckert.
* Added vendor gem support to RubyGems.Package managers may now install gems
in Gem.vendor_dir with the --vendor option to gem install.Issue #943 by
Marcus R眉ckert.
Bug fixes:
* Kernel#gem now respects the prerelease flag when activating gems.
Previously this behavior was undefined which could lead to bugs when a
prerelease version was unintentionally activated.Bug #938 by Joe Ferris.
* RubyGems now prefers gems from git over installed gems.This allows gems
from git to override an installed gem with the same name and version.Bug
#944 by Thomas Kriechbaumer.
* Fixed handling of git gems in a lockfile with unversioned dependencies.Bug
#940 by Michael Kaiser-Nyman.
* The ruby directive in a gem dependencies file is ignored when installing.
Bug #941 by Michael Kaiser-Nyman.
* Added open to list of builtin commands (`gem open` now works).Reported by
Espen Antonsen.
* `gem open` now works with command-line editors.Pull request #962 by Tim
Pope.
* `gem install -g` now respects `--conservative`.Pull request #950 by Jeremy
Evans.
* RubyGems> Alexander E. Fischer.
* RubyGems now expands ~ in $PATH when checking if installed executables will
be runnable.Pull request #945 by Alex Talker.
* Fixed `gem install -g --explain`.Issue #947 by Luis Lavena.Patch by
Hsing-Hui Hsu.
* RubyGems locks less during gem activation.Pull request #951 by Aaron
Patterson and Justin Searls, #969 by Jeremy Tryba.
* Kernel#gem is now thread-safe.Pull request #967 by Aaron Patterson.
* RubyGems now handles spaces in directory names for some parts of extension
building.Pull request #949 by Tristan Hill.
* RubyGems no longer defines an empty Date> Daloze.
* RubyGems respects --document options for `gem update` again.Bug 946 by
jonforums.Patch by Hsing-Hui Hsu.
* RubyGems generates documentation again with --ignore-dependencies.Bug #961
by Pulfer.
* RubyGems can install extensions across partitions now.Pull request #970 by
Michael Scherer.
* `-s` is now short for `--source` which resolves an ambiguity with
--no-suggestions.Pull request #955 by Alexander Kahn.
* Added extra test for ~> for 0.0.X versions.Pull request #958 by Mark
Lorenz.
* Fixed typo in gem updated help.Pull request #952 by Per Modin.
* Clarified that the gem description should not be excessively long.Part of
bug #956 by Renier Morales.
* Hid documentation of outdated test_files> Guides issue #90 by Emil Soman.
* RubyGems now falls back to the old index if the rubygems.org API fails
during gem resolution.
------------------------------------------------------------------------------
RubyGems installed the following executables:
/usr/bin/gem
Ruby Interactive (ri) documentation was installed. ri is kind of like man
pages for ruby libraries. You may access it like this:
ri>
ri>
ri> If you do not wish to install this documentation in the future, use the
--no-document flag, or set it as the default in your ~/.gemrc file. See
'gem help env' for details.
为了方便操作rubygem，我们把它拷贝到系统环境变量当中，方便调用
cp bin/gem /usr/local/bin/ 17、接下来，安装 redis 的api 接口，这就是为什么上面要装rubygem的道理了
gem install -l redis-3.0.0.gem redis-3.0.0.gem上面讲到，可以到 官网http://rubygems.org/gems/redis/versions/3.1.0下载
18、一切就绪，现在再一次，来执行集群看看还有没有报错的信息
# ./redis-trib.rb create --replicas 1 192.168.1.222:6379 192.168.1.222:6380 192.168.1.222:6381 192.168.1.223:6382 192.168.1.223:6383 192.168.1.223:6384
>>> Creating cluster
Connecting to node 192.168.1.222:6379: OK
Connecting to node 192.168.1.222:6380: OK
Connecting to node 192.168.1.222:6381: OK
Connecting to node 192.168.1.223:6382: OK
Connecting to node 192.168.1.223:6383: OK
Connecting to node 192.168.1.223:6384: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.1.223:6382
192.168.1.222:6379
192.168.1.223:6383
Adding replica 192.168.1.222:6380 to 192.168.1.223:6382
Adding replica 192.168.1.223:6384 to 192.168.1.222:6379
Adding replica 192.168.1.222:6381 to 192.168.1.223:6383
M: faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 192.168.1.222:6379
slots:5461-10922 (5462 slots) master
S: 92999f9840418a848f7b10c5bca0119e3b515fa4 192.168.1.222:6380
replicates c13fbd7fadda1e46b133b8f87248414d9a7a0fcc
S: 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b 192.168.1.222:6381
replicates 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5
M: c13fbd7fadda1e46b133b8f87248414d9a7a0fcc 192.168.1.223:6382
slots:0-5460 (5461 slots) master
M: 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 192.168.1.223:6383
slots:10923-16383 (5461 slots) master
S: c4b565e9236effee3957c6f03b533dced729e39b 192.168.1.223:6384
replicates faa2fab465017c9be2475d0cbd9ffb7b8e8267e6
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.1.222:6379)
M: faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 192.168.1.222:6379
slots:5461-10922 (5462 slots) master
M: 92999f9840418a848f7b10c5bca0119e3b515fa4 192.168.1.222:6380
slots: (0 slots) master
replicates c13fbd7fadda1e46b133b8f87248414d9a7a0fcc
M: 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b 192.168.1.222:6381
slots: (0 slots) master
replicates 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5
M: c13fbd7fadda1e46b133b8f87248414d9a7a0fcc 192.168.1.223:6382
slots:0-5460 (5461 slots) master
M: 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 192.168.1.223:6383
slots:10923-16383 (5461 slots) master
M: c4b565e9236effee3957c6f03b533dced729e39b 192.168.1.223:6384
slots: (0 slots) master
replicates faa2fab465017c9be2475d0cbd9ffb7b8e8267e6
All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
All 16384 slots covered.
#
19、很明显，全部正常，一切OK，搞定。
20、如果不太相信，那么你大可以来测试一下
# ./redis-trib.rb check 192.168.1.223:6384
Connecting to node 192.168.1.223:6384: OK
Connecting to node 192.168.1.222:6380: OK
Connecting to node 192.168.1.222:6379: OK
Connecting to node 192.168.1.222:6381: OK
Connecting to node 192.168.1.223:6383: OK
Connecting to node 192.168.1.223:6382: OK
>>> Performing Cluster Check (using node 192.168.1.223:6384)
S: c4b565e9236effee3957c6f03b533dced729e39b 192.168.1.223:6384
slots: (0 slots) slave
replicates faa2fab465017c9be2475d0cbd9ffb7b8e8267e6
S: 92999f9840418a848f7b10c5bca0119e3b515fa4 192.168.1.222:6380
slots: (0 slots) slave
replicates c13fbd7fadda1e46b133b8f87248414d9a7a0fcc
M: faa2fab465017c9be2475d0cbd9ffb7b8e8267e6 192.168.1.222:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b 192.168.1.222:6381
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 7d674df0ab68be0b4c6a461b0c1484b02eeb12c5 192.168.1.223:6383
slots: (0 slots) slave
replicates 5d44e7e3bf2c19fcdcc1548feef42d5d9665da0b
M: c13fbd7fadda1e46b133b8f87248414d9a7a0fcc 192.168.1.223:6382
slots:0-5460 (5461 slots) master
1 additional replica(s)
All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
All 16384 slots covered.
#
注意哦，我是直接在192.168.1.222，测试到192.168.1.223当中的6384这个实例了
从上面，我们也不难看出
redis_master 是
192.168.1.222:6379
192.168.1.222:6381
192.168.1.223:6382
redis_salve 是
192.168.1.223:6384
192.168.1.222:6380
192.168.1.223:6383
21、实验结果
任何一个实例，都能写入，查询，不存在主备关系，redis会自动通过内部算法，选举，不需要你去关心
22、做一个小测试
在 192.168.1.222 ，连接 192.168.1.223 的实例D 6382 ，写入数据 zhou
# redis-cli -h 192.168.1.223 -p 6382 -c
192.168.1.223:6382> set zhou mynameiszhou
-> Redirected to slot located at 192.168.1.223:6383
OK
Redis会自己重定向到 6383 ，也就是 E 当中，换句话说，6382和6382是主备关系，其中6383是6382的 salver 服务器
好，那我们来 192.168.1.223 里面的 6484，也就是 F当中读取 zhou的值
# redis-cli -h 192.168.1.223 -p 6384 -c
192.168.1.223:6384> get zhou
-> Redirected to slot located at 192.168.1.223:6383
"mynameiszhou"
很明显，读取到了，它 会自动映射到 6383 去读取
现在疑问来了，我就把6383干掉，看看是什么样的情况
# ps aux |grep redis
root 289220.20.9 1374089752 pts/2 Sl 03:30 0:06 redis-server *:6382
root 289280.20.9 1374089596 pts/2 Sl 03:31 0:06 redis-server *:6383
root 289320.20.7 1374087884 pts/2 Sl 03:32 0:05 redis-server *:6384
root 289730.00.0 103232 876 pts/2 S+ 04:18 0:00 grep redis
# kill -9 28928
# ps aux |grep redis
root 289220.20.9 1374089752 pts/2 Sl 03:30 0:06 redis-server *:6382
root 289320.20.7 1374087884 pts/2 Sl 03:32 0:05 redis-server *:6384
root 289750.00.0 103232 876 pts/2 S+ 04:19 0:00 grep redis
-Killed redis-server /opt/redis-6383.conf > /opt/redis-6383.log 2>&1
很明显，6383已经被干掉了
# redis-cli -h 192.168.1.223 -p 6383 -c
Could not connect to Redis at 192.168.1.223:6383: Connection refused
not connected> get zhou
Could not connect to Redis at 192.168.1.223:6383: Connection refused
not connected>
# redis-cli -h 192.168.1.223 -p 6384 -c
192.168.1.223:6384> get zhou
-> Redirected to slot located at 192.168.1.222:6381
"mynameiszhou"
192.168.1.222:6381>
虽然，在6383，建立不了链接，但是不影响我数据的读取，在6381一样可以获取到zhou的key值
very good，perfect!!!
页:
[1]