Memcached+Magent+Keepalived高可用架构部署
Memcached集群介绍[*]现实生产环境中，我们要实现memcached主从复制避免单点故障风险，往往需要部署高可用群集。这里我们使用magent代理memcached实现主从备份，从而来保证缓存数据的完整性。工作原理如下：
1）数据在到达memcached服务器前会经过magent，每次写数据到主memcached服务器，magent会同时写到从 服务器 上，并且主从服务器写的算法一样。
2）当主memcached服务器宕掉之后，magent会向从memcached服务器中读取数据
3）当memcached恢复后，magent将重新从主memcached中读取数据，此时由于主memcached刚恢复正常，所以在宕掉期间的数据是无法获取的，这也是magent的缺点
[*]再通过keepalived设置VIP地址，提供给Memcached API客户端以及magent连接访问。
1）memcached主从复制这种架构，在程序连接时不知道应该连接哪个主服务器，所以需要用keepalived在前端加VIP地址实现高可用；
2）keepalived不断检测memcached主服务器的11211端口，如果检测到memcached服务发生宕机或者死机等情况，就会将VIP从主服务器移至从服务器，从而实现高可用。
Memcached高可用架构图：
http://i2.运维网.com/images/blog/201809/26/3093d3327aec822f69d3a719c4534801.png
实验环境介绍：
服务器主机
IP地址
需要安装的软件包
操作系统
master
192.168.30.55
VIP:192.168.30.16
libevent-2.1.8-stable.tar.gz，memcached-1.5.6.tar.gz,
magent-0.5.tar.gz，keepalived
CentOS7
backup
192.168.30.10
VIP:192.168.30.16
libevent-2.1.8-stable.tar.gz，memcached-1.5.6.tar.gz，
keepalived
CentOS7
client
192.168.30.15
telnet
RedHat6.5
百度网盘免费提供软件包：
libevent-2.1.8-stable.tar.gz：https://pan.baidu.com/s/1TBVEyndXVHcMbxst-kpCuA
memcached-1.5.6.tar.gz：https://pan.baidu.com/s/1oO_mC6J3SV6PLVepjmi2eQ
magent-0.5.tar.gz：https://pan.baidu.com/s/1xoT3Px14RHu6QE-66nLtMQ
实验部署：
首先所有主机关闭防火墙，及selinux
systemctl stop firewalld.service（CentOS7上）或者service iptables stop（RedHat6.5上）
setenforce 0
配置memcached主缓存节点和从缓存节点（主，从基本步骤相同，个别不同步骤下面会单独说明）
[*]将事先准备的软件包进行解压
tar xvf memcached-1.5.6.tar.gz -C /opt/
tar xvf libevent-2.1.8-stable.tar.gz -C /opt/#事件通知库
[*]安装编译环境包
yum install gcc gcc-c++ make -y
编译安装libvent
cd /opt/libevent-2.1.8-stable
./configure --prefix=/usr #指定工作目录
make && make install
[*]编译安装memcached
cd /opt/memcached-1.5.6
./configure \
--prefix=/usr/local/memcached \ #指定工作目录
--with-libevent=/usr #指定libevent的安装路径
make && make install
ln -s /usr/local/memcached/bin/* /usr/local/bin/#让系统识别命令
[*]在主服务器安装magent，从服务器不用安装
mkdir /opt/magent
cd ～/
tar xvf magent-0.5.tar.gz -C /opt/magent/
ketama.c
magent.c
ketama.h
Makefile
cd /opt/magent/
编辑配置文件
vim ketama.h
#ifndef SSIZE_MAX #将原来的参数修改为SSIZE_MAX
#define SSIZE_MAX 32767 #将原来的参数修改为SSIZE_MAX 32767
http://i2.运维网.com/images/blog/201809/26/22d0bea513ad6c9c71291c818d17f7f2.png
修改Makefile
vim Makefile
LIBS = -levent -lm #在后面加上-lm
http://i2.运维网.com/images/blog/201809/26/246857e13ba80f973e61d80a80eec8a6.png
编译（完成后会生成一个magent的可执行文件）
make
http://i2.运维网.com/images/blog/201809/26/a9dec821923e2d3d04e9e1d9ecad5734.png
复制mgent命令到系统管理中，便于使用
cp magent /usr/bin/#将magent可执行文件加入usr可执行命令中
[*]把产生的magent文件远程复制到从服务器上
#scp magent 192.168.30.10:/usr/bin
The authenticity of host '192.168.30.10 (192.168.30.10)' can't be established.
ECDSA key fingerprint is SHA256:/5yAWf89joXesnoqczcLtVetMK2DIseKpZOcWkC+7V4.
ECDSA key fingerprint is MD5:a1:a4:ef:ff:84:46:0b:0c:fb:1d:de:56:4e:36:89:15.
Are you sure you want to continue connecting (yes/no)?yes #输入yes，同意复制
Warning: Permanently added '192.168.30.10' (ECDSA) to the list of known hosts.
root@192.168.30.10's password:#输入从服务器的root密码
magent 100%112KB 4.9MB/s 00:00
[*]开启memcached缓存数据库（主，从都开启）
memcached -d -m 32m -p 11211 -u root
[*]检查memcached进程是否运行
netstat -antp | grep 11211
tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 57782/memcached
tcp6 0 0 :::11211 :::* LISTEN 57782/memcached
接着在主从服务器上安装配置keepalived
yum install keepalived -y
[*]修改配置文件keepalived.conf
vim /etc/keepalived/keepalived.conf
其余设置项删除掉，就留下面的设置
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id MAGENT_HA #指定router_id， 从服务器为MAGENT_HB
}
vrrp_script magent { #定义函数
script "/opt/shell/magent.sh"
interval 2
}
vrrp_instance VI_1 {
state MASTER #从服务器为BACKUP
interface ens33 #本地网卡名称
virtual_router_id 51 #从服务器id不同
priority 100 #从服务器优先级要小于主的
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script { #调用函数
magent
}
virtual_ipaddress {
192.168.30.16 #设置VIP
}
[*]主服务器上的keepalived配置
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id MAGENT_HA
}
vrrp_script magent {
script "/opt/shell/magent.sh"
interval 2
}
vrrp_instance VI_1 {
stateMASTER
interface ens33
virtual_router_id 51
priority100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
magent
}
virtual_ipaddress {
192.168.30.16
}
}
[*]从服务器上的keepalived配置
# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_idMAGENT_HB
}
vrrp_script magent {
script "/opt/shell/magent.sh"
interval 2
}
vrrp_instance VI_1 {
stateBACKUP
interfaceens33
virtual_router_id 52
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
magent
}
virtual_ipaddress {
192.168.30.16
}
}
在主服务器上建立/opt/shell目录，创建magent.sh脚本
mkdir /opt/shell
vim /opt/shell/magent.sh
#!/bin/bash
K=`ps -ef | grep keepalived | grep -v grep | wc -l`
if [ $K -gt 0 ]; then
magent -u root -n 51200 -l 192.168.30.16 -p 12000 -s 192.168.30.55:11211 -b 192.168.30.10:11211
else
pkill -9 magent
fi
[*]其中的参数说明
-n 51200 //定义用户最大连接数
-l 192.168.175.188 //指定虚拟IP
-p 12000//指定端口号
-s //指定主缓存服务器
-b //指定从缓存服务器
[*]赋予脚本执行权限
chmod +x magent.sh
[*]启动keepalived服务
systemctl start keepalived.service
[*]查看12000端口的进程（magent）是否运行
netstat -antp | grep 12000
tcp 0 0 192.168.30.16:12000 0.0.0.0:* LISTEN 22180/magent
[*]查看系统公共日志文件/var/log/messages ，检测keepalived功能是否启用
vim /var/log/messages
Sep 26 12:20:42 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 26 12:20:43 localhost Keepalived_vrrp: VRRP_Instance(VI_1)Entering MASTER STATE
Sep 26 12:20:43 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 26 12:20:43 localhost Keepalived_vrrp: Sending gratuitous ARP on ens33 for 192.168.30.16
Sep 26 12:20:43 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.30.16
同样在从服务器上建立/opt/shell目录，创建magent.sh脚本
mkdir /opt/shell
vim /opt/shell/magent.sh
#!/bin/bash
K=`ip addr | grep 192.168.30.16 | grep -v grep | wc -l`
if [ $K -gt 0 ]; then
magent -u root -n 51200 -l 192.168.30.16 -p 12000 -s 192.168.30.55:11211 -b 192.168.30.10:11211
else
pkill -9 magent
fi
[*]赋予脚本执行权限
chmod +x magent.sh
[*]启动keepalived服务
systemctl start keepalived.service
[*]查看12000端口的进程（magent）是否运行
netstat -antp | grep 12000
tcp 0 0 192.168.30.16:12000 0.0.0.0:* LISTEN 22180/magent
[*]查看系统公共日志文件/var/log/messages ，检测keepalived功能是否启用
vim /var/log/messages
Sep 26 12:32:50 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 26 12:32:50 localhost Keepalived_vrrp: VRRP sockpool:
Sep 26 12:32:51 localhost Keepalived_vrrp: VRRP_Script(magent) succeeded
在client上测试
[*]安装telnet，
yum -y install telnet
[*]连接VIP，端口12000进行远程登录memcached群集，测试主从复制同步，添加键值username
# telnet 192.168.30.16 12000
Trying 192.168.30.16...
Connected to 192.168.30.16.
Escape character is '^]'.
add username 0 0 6
abcdef
STORED
quit
Connection closed by foreign host.
[*]连接master主服务器，查看username，同步成功
#telnet 192.168.30.55 11211
Trying 192.168.30.55...
Connected to 192.168.30.55.
Escape character is '^]'.
get username
VALUE username 0 6
abcdef
END
quit
Connection closed by foreign host.
[*]连接backup从服务器，查看username，同步成功
#telnet 192.168.30.10 11211
Trying 192.168.30.10...
Connected to 192.168.30.10.
Escape character is '^]'.
get username
VALUE username 0 6
abcdef
END
quit
Connection closed by foreign host.
测试故障转移
[*]在主服务器端关闭memcached服务
http://i2.运维网.com/images/blog/201809/26/4af309de2376203df74259813478e5e7.jpg
[*]确认从服务器端memcached服务正常运行
http://i2.运维网.com/images/blog/201809/26/48fd79a4bfdc07b89ea037e4155eb91d.jpg
[*]在client端通过VIP连接群集，检测群集仍然正常运行
http://i2.运维网.com/images/blog/201809/26/54b835d5d487452d5f3dd7efe5c428fa.jpg
页:
[1]