Heartbeat+Drbd+MySQL高可用

ooioiiuu · 发表于 2016-9-28 09:00:12

一、环境介绍

继续使用之前heartbeat+drbd+nfs的环境，192.168.49.0/24网段用来ssh远程连接，172.16.49.0/24用来做心跳连接，并且也做drbd同步数据使用。因为中间做了好多改变，这里再次给出环境的配置情况。

主机名	角色	IP地址
heartbeat01.contoso.com	heartbeat+drbd+mysql(节点1）	eth0:192.168.49.133 eth1:172.16.49.133
heartbeat02.contoso.com	heartbeat+drbd+mysql（节点2）	eth0:192.168.49.134 eth1:172.16.49.134

关于heartbeat和drbd的安装就不再描述了，之前的博文中有，下面只是贴出来之前两者的相关配置：
1、heartbeat的配置文件内容（下面以heartbeat02为例来列举，heartbeat01上的配置与heartbeat02相同）如下：
authkeys的内容：
[iyunv@heartbeat02 ~]# egrep -v "#|^$" /etc/ha.d/authkeys
auth 2
2 sha1 c6091592594cd14c
ha.cf的内容：
[iyunv@heartbeat02 ~]# egrep -v "#|^$" /etc/ha.d/ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local1
keepalive 2
deadtime 30
warntime 10
initdead 60
udpport 694
mcast eth1 225.0.0.1 694 1 0
auto_failback on
node heartbeat01.contoso.com
node heartbeat02.contoso.com
respawn hacluster /usr/lib64/heartbeat/ipfail
haresources的内容：
[iyunv@heartbeat02 ~]# egrep -v "#|^$" /etc/ha.d/haresources
heartbeat01.contoso.com  IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 nfs killnfsd  #这里是完成nfs高可用后的配置
2、drbd的配置文件内容（下面也是以heartbeat02为例，heartbeat01和heartbeat02的配置相同）如下：
[iyunv@heartbeat02 ~]# cat /etc/drbd.d/test.res
resource test
{
startup {
wfc-timeout 30;
outdated-wfc-timeout 20;
degr-wfc-timeout 30;
}
net {
cram-hmac-alg sha1;
shared-secret sync_disk;
}
syncer {
rate 200M;
al-extents 257;
on-no-data-accessible io-error;
}
on heartbeat01.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 172.16.49.133:7788;
flexible-meta-disk internal;
}
on heartbeat02.contoso.com {
device /dev/drbd0;
disk /dev/sdb1;
address 172.16.49.134:7788;
meta-disk internal;
}
}
3、另外需要注意的一些点

1）编辑hosts文件，添加对应主机的解析
[iyunv@heartbeat02 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1       localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.49.133 heartbeat01.contoso.com heartbeat01
172.16.49.134 heartbeat02.contoso.com heartbeat02
2）添加时间同步定时任务和主机路由
[iyunv@heartbeat02 ~]# tail -2 /etc/rc.local
/usr/sbin/ntpdate  time.nist.gov 210.72.145.44 64.147.116.229  #校准主机时间
route add -host 172.16.49.133 dev eth1 #两台主机都要添加一条主机路由，主机都是对方的ip
3）关闭iptables并禁用SELinux
[iyunv@heartbeat02 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target    prot opt source             destination

Chain FORWARD (policy ACCEPT)
target    prot opt source             destination

Chain OUTPUT (policy ACCEPT)
target    prot opt source             destination
[iyunv@heartbeat02 ~]# getenforce
Disabled
4）查看一下开机启动项
[iyunv@heartbeat02 ~]# chkconfig --list|grep heartbeat
heartbeat       0:off 1:off 2:off 3:off 4:off 5:off 6:off
[iyunv@heartbeat02 ~]# chkconfig --list|grep drbd
drbd            0:off 1:off 2:on 3:on 4:on 5:on 6:off
[iyunv@heartbeat02 ~]# chkconfig --list|grep rpcbind
rpcbind         0:off 1:off 2:on 3:on 4:on 5:on 6:off
[iyunv@heartbeat02 ~]# chkconfig --list|grep nfs
nfs             0:off 1:off 2:off 3:off 4:off 5:off 6:off
二、安装MySQL
这里网上有很多教程，有的是rpm包的方式安装，然后使用脚本初始化，有的可参考老男孩的脚本安装，而我这里直接使用yum安装MySQL，然后通过修改配置文件的方式，让MySQL的数据保存在drbd同步目录(/data)中。
1、节点1（heartbeat01）上MySQL的安装及配置
1）yum安装MySQL并测试本地启动
[iyunv@heartbeat01 ~]# yum -y install mysql-devel mysql-server
[iyunv@heartbeat01 ~]# /etc/init.d/mysqld start
Initializing MySQL database:  Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h heartbeat01.contoso.com password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!

                                                         [  OK  ]
Starting mysqld:                                        [  OK  ]
[iyunv@heartbeat01 ~]# mysqladmin -u root password "123456"
[iyunv@heartbeat01 ~]# mysql -uroot -p123456
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> quit
Bye
2）将heartbeat01作为drbd的Primary节点，然后挂载到/data下
[iyunv@heartbeat01 lib]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat01 lib]# drbdadm -- --overwrite-data-of-peer primary test
[iyunv@heartbeat01 lib]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat01 lib]# mount /dev/drbd0 /data
[iyunv@heartbeat01 data]# rm -rf *  #清空之前nfs同步的文件
3）编辑MySQL配置文件
[iyunv@heartbeat01 ~]# cp /etc/my.cnf /etc/my.cnf.bak$(date +%F)
[iyunv@heartbeat01 ~]# vi /etc/my.cnf
[iyunv@heartbeat01 ~]# diff /etc/my.cnf.bak2016-09-26 /etc/my.cnf
2,3c2,3
< datadir=/var/lib/mysql
< socket=/var/lib/mysql/mysql.sock
---
> datadir=/data/lib/mysql
> socket=/data/lib/mysql/mysql.sock
9,10c9,10
< log-error=/var/log/mysqld.log
< pid-file=/var/run/mysqld/mysqld.pid
---
> log-error=/data/log/mysqld.log
> pid-file=/data/run/mysqld/mysqld.pid
[iyunv@heartbeat01 ~]# cp /etc/init.d/mysqld /etc/init.d/mysqld.bak$(date +%F)
[iyunv@heartbeat01 ~]# vi /etc/init.d/mysqld
[iyunv@heartbeat01 ~]# diff /etc/init.d/mysqld.bak2016-09-26 /etc/init.d/mysqld
35c35
< lockfile=/var/lock/subsys/$prog
---
> lockfile=/data/lock/subsys/$prog
51c51
< get_mysql_option mysqld datadir "/var/lib/mysql"
---
> get_mysql_option mysqld datadir "/data/lib/mysql"
55c55
< get_mysql_option mysqld_safe log-error "/var/log/mysqld.log"
---
> get_mysql_option mysqld_safe log-error "/data/log/mysqld.log"
57c57
< get_mysql_option mysqld_safe pid-file "/var/run/mysqld/mysqld.pid"
---
> get_mysql_option mysqld_safe pid-file "/data/run/mysqld/mysqld.pid"
[iyunv@heartbeat01 ~]# cp /usr/bin/mysqld_safe /usr/bin/mysqld_safe.bak$(date +%F)
[iyunv@heartbeat01 ~]# vi /usr/bin/mysqld_safe
[iyunv@heartbeat01 ~]# diff /usr/bin/mysqld_safe.bak2016-09-27 /usr/bin/mysqld_safe
286c286
< DATADIR=/var/lib/mysql
---
> DATADIR=/data/lib/mysql
436c436
< safe_mysql_unix_port=${mysql_unix_port:-${MYSQL_UNIX_PORT:-/var/lib/mysql/mysql.sock}}
---
> safe_mysql_unix_port=${mysql_unix_port:-${MYSQL_UNIX_PORT:-/data/lib/mysql/mysql.sock}}
4）拷贝MySQL目录文件
[iyunv@heartbeat01 ~]# mkdir -p /data/{lib,run,lock,log}
[iyunv@heartbeat01 ~]# cp -ar /var/lib/mysql /data/lib/
[iyunv@heartbeat01 ~]# cp -ar /var/run/mysqld /data/run/
[iyunv@heartbeat01 ~]# cp -ar /var/log/mysqld.log /data/log
[iyunv@heartbeat01 ~]# chown -R mysql:mysql /data/lib
[iyunv@heartbeat01 ~]# chown -R mysql:mysql /data/run
[iyunv@heartbeat01 ~]# chown -R mysql:mysql /data/log
[iyunv@heartbeat01 ~]# chown -R mysql:mysql /data/lock
[iyunv@heartbeat01 /]# ln -s /data/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock
#这一条命令是防止使用mysql命令时找不到/var/lib/mysql/mysql.sock文件
5）测试MySQL能否在/data下启动
[iyunv@heartbeat01 /]# /etc/init.d/mysqld start
Starting mysqld:                                        [  OK  ]
[iyunv@heartbeat01 /]# mysql -uroot -p123456
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database          |
+--------------------+
| information_schema |
| mysql             |
| test             |
+--------------------+
3 rows in set (0.00 sec)
2、节点2（heartbeat02）上MySQL的安装及配置
1）安装MySQL
[iyunv@heartbeat02 ~]# yum -y install mysql-devel mysql-server
2）配置MySQL

在节点2上不需要拷贝MySQL相关文件，因为相关文件已经在heartbeat01上拷贝到drbd的同步目录中了，也已经通过drbd同步到heartbeat02上了，所以这里就不需要拷贝目录了，只需要修改配置文件即可，当然，可以使用scp命令拷贝heartbeat01上修改好的配置文件。
[iyunv@heartbeat02 ~]# mv /etc/my.cnf /etc/my.cnf.bak
[iyunv@heartbeat02 ~]# scp heartbeat01:/etc/my.cnf /etc/my.cnf
root@heartbeat01's password:
my.cnf
[iyunv@heartbeat02 ~]# mv /etc/init.d/mysqld /etc/init.d/mysqld.bak
[iyunv@heartbeat02 ~]# scp heartbeat01:/etc/init.d/mysqld /etc/init.d/mysqld
root@heartbeat01's password:
mysqld
[iyunv@heartbeat02 ~]# mv /usr/bin/mysqld_safe /usr/bin/mysqld_safe.bak
[iyunv@heartbeat02 ~]# scp heartbeat01:/usr/bin/mysqld_safe /usr/bin/mysqld_safe
root@heartbeat01's password:
mysqld_safe                                                    [iyunv@heartbeat02 ~]# ln -s /data/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock
3）测试heartbeat02上的MySQL能否在/data下启动
要测试heartbeat02上的MySQL能否在drbd的同步目录下启动，就要把drbd的目录挂载到heartbeat02的/data目录下。
[iyunv@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat02 ~]# drbdadm disconnect test
[iyunv@heartbeat02 ~]# drbdadm connect test
[iyunv@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat02 ~]# drbdadm primary test
[iyunv@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat02 ~]# mount /dev/drbd0 /data
[iyunv@heartbeat02 ~]# ll /data
total 16
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 lib
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:21 lock
drwxr-xr-x 2 mysql mysql 4096 Sep 27 01:18 log
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 run
[iyunv@heartbeat02 ~]# /etc/init.d/mysqld start
Starting mysqld:                                        [  OK  ]
[iyunv@heartbeat02 ~]# mysql -uroot -p123456
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database          |
+--------------------+
| information_schema |
| mysql             |
| test             |
+--------------------+
3 rows in set (0.02 sec)

mysql> select Host,User,Password from mysql.user;
+-------------------------+------+-------------------------------------------+
| Host                   | User | Password                               |
+-------------------------+------+-------------------------------------------+
| localhost             | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
| heartbeat01.contoso.com | root |                                        |
| 127.0.0.1             | root |                                        |
| localhost             |    |                                        |
| heartbeat01.contoso.com |    |                                        |
+-------------------------+------+-------------------------------------------+
5 rows in set (0.00 sec)

mysql> quit
Bye
注意：只有在两个节点上手工启动MySQL成功后，才可以使用heartbeat做MySQL的高可用。上面的步骤成功后，开始在heartbeat中配置MySQL服务。
三、heartbeat实现MySQL高可用
1、检查节点的状态
[iyunv@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:24 dw:24 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat01 ~]# /etc/init.d/heartbeat status
heartbeat is stopped. No process
[iyunv@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat02 ~]#  /etc/init.d/heartbeat status
heartbeat is stopped. No process
2、修改heartbeat的haresources文件
[iyunv@heartbeat01 ~]# vi /etc/ha.d/haresources
[iyunv@heartbeat01 ~]# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.com  IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 mysqld
[iyunv@heartbeat02 ~]# vi /etc/ha.d/haresources
[iyunv@heartbeat02 ~]# tail -1 /etc/ha.d/haresources
heartbeat01.contoso.com  IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 mysqld
3、开启heartbeat服务，查看MySQL服务是否正常
[iyunv@heartbeat01 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:  Resource is stopped
Done.
[iyunv@heartbeat02 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO:  Resource is stopped
Done.
4、在heartbeat01上进行检查
[iyunv@heartbeat01 ~]# ip a|grep 49.100
inet 192.168.49.100/24 scope global eth1
[iyunv@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:192 nr:24 dw:216 dr:3929 al:7 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat01 ~]# ll /data
total 16
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 lib
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:21 lock
drwxr-xr-x 2 mysql mysql 4096 Sep 27 01:18 log
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 run
[iyunv@heartbeat01 ~]# /etc/init.d/mysqld status
mysqld (pid  10360) is running...
[iyunv@heartbeat01 ~]# mysql -uroot -p123456
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select Host,User,Password from mysql.user;
+-------------------------+------+-------------------------------------------+
| Host                   | User | Password                               |
+-------------------------+------+-------------------------------------------+
| localhost             | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
| heartbeat01.contoso.com | root |                                        |
| 127.0.0.1             | root |                                        |
| localhost             |    |                                        |
| heartbeat01.contoso.com |    |                                        |
+-------------------------+------+-------------------------------------------+
5 rows in set (0.00 sec)

mysql>
5、在heartbeat02上进行检查
[iyunv@heartbeat02 ~]# ip a|grep 49.100
[iyunv@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:208 dw:208 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat02 ~]# ll /data
total 0
[iyunv@heartbeat02 ~]# /etc/init.d/mysqld status
mysqld is stopped
[iyunv@heartbeat02 ~]#
四、进行故障切换测试
1、在heartbeat01上创建一个数据库，看是否能同步到heartbeat02上
mysql> create database tb01;
Query OK, 1 row affected (0.02 sec)
mysql> use tb01;
Database changed
mysql> create table staff(id int primary key, name varchar2(40) null);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'varchar2(40) null)' at line 1
mysql> create table staff (
-> id int not null,
-> name varchar(40) null,
-> primary key (id));
Query OK, 0 rows affected (0.14 sec)

mysql> insert into staff values (01,'Tom');
Query OK, 1 row affected (0.01 sec)

mysql> insert into staff values (02,'Jerry');
Query OK, 1 row affected (0.00 sec)

mysql> select * from staff;
+----+-------+
| id | name  |
+----+-------+
|  1 | Tom |
|  2 | Jerry |
+----+-------+
2 rows in set (0.00 sec)


mysql> quit
Bye
2、停止heartbeat01上的heartbeat服务，查看是否MySQL能切换到heartbeat02并运行正常
[iyunv@heartbeat01 ~]# /etc/init.d/heartbeat stop
Stopping High-Availability services: Done.
1）在heartbeat01上
[iyunv@heartbeat01 ~]# ip a |grep 49.100
[iyunv@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:708 nr:264 dw:972 dr:3957 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat01 ~]# ll /data
total 0
[iyunv@heartbeat01 ~]# /etc/init.d/mysqld status
mysqld dead but subsys locked  #这里是因为无法删除MySQL锁定文件所致，但不影响这里的测试
解决方法：
[iyunv@heartbeat01 ~]# mv /var/lock/subsys/mysqld /tmp
[iyunv@heartbeat01 ~]# /etc/init.d/mysqld status
mysqld is stopped
当然，也可以在/etc/init.d/mysqld脚本中的stop方法中加入“rm -f /var/lock/subsys/mysqld”一行即可。
2）在heartbeat02上
[iyunv@heartbeat02 ~]# ip a|grep 49.100
inet 192.168.49.100/24 scope global eth1
[iyunv@heartbeat02 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:188 nr:708 dw:896 dr:3929 al:6 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat02 ~]# ll /data
total 16
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 lib
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:21 lock
drwxr-xr-x 2 mysql mysql 4096 Sep 27 01:18 log
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 run
[iyunv@heartbeat02 ~]# /etc/init.d/mysqld status
mysqld (pid  8578) is running...
[iyunv@heartbeat02 ~]# mysql -uroot -p123456
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from tb01.staff;
+----+-------+
| id | name  |
+----+-------+
|  1 | Tom |
|  2 | Jerry |
+----+-------+
2 rows in set (0.04 sec)

mysql> quit
Bye

OK，可以看到，不仅heartbeat02上成功启动了MySQL服务，而且之前在heartbeat01上创建的数据库和表都已经同步过来。
3、重新启动heartbeat01的heartbeat服务，看一下heartbeat01是否能接管MySQL服务

[iyunv@heartbeat01 ~]# ip a|grep 49.100
inet 192.168.49.100/24 scope global eth1
[iyunv@heartbeat01 ~]# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:884 nr:448 dw:1332 dr:7886 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[iyunv@heartbeat01 ~]# ll /data
total 16
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 lib
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:21 lock
drwxr-xr-x 2 mysql mysql 4096 Sep 27 01:18 log
drwxr-xr-x 3 mysql mysql 4096 Sep 27 01:17 run
[iyunv@heartbeat01 ~]# /etc/init.d/mysqld status
mysqld (pid  11668) is running...
可以看到，heartbeat01成功接管了所有的资源和服务，MySQL也已经在heartbeat01上运行了。

[iyunv@heartbeat02 ~]# /etc/init.d/mysqld status
mysqld is stopped
另外，heartbeat02上的MySQL服务已经停止，没有出现mysqld dead but subsys locked的问题了。
4、查看故障切换中的heartbeat日志
[iyunv@heartbeat01 ~]# cat /var/log/ha-log
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10991]: info: Pacemaker support: false
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10991]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10991]: info: **************************
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10991]: info: Configuration validated. Starting heartbeat 3.0.4
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10992]: info: heartbeat: version 3.0.4
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10992]: info: Heartbeat generation: 1474533059
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10992]: info: glib: UDP multicast heartbeat started for group 225.0.0.1 port 694 interface eth1 (ttl=1 loop=0)
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10992]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10992]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10992]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Sep 27 05:56:50 heartbeat01.contoso.com heartbeat: [10992]: info: Local status now set to: 'up'
Sep 27 05:56:52 heartbeat01.contoso.com heartbeat: [10992]: info: Link heartbeat02.contoso.com:eth1 up.
Sep 27 05:56:52 heartbeat01.contoso.com heartbeat: [10992]: info: Status update for node heartbeat02.contoso.com: status active
harc(default)[11000]: 2016/09/27_05:56:52 info: Running /etc/ha.d//rc.d/status status
Sep 27 05:56:53 heartbeat01.contoso.com heartbeat: [10992]: info: Comm_now_up(): updating status to active
Sep 27 05:56:53 heartbeat01.contoso.com heartbeat: [10992]: info: Local status now set to: 'active'
Sep 27 05:56:53 heartbeat01.contoso.com heartbeat: [10992]: info: Starting child client "/usr/lib64/heartbeat/ipfail" (498,499)
Sep 27 05:56:53 heartbeat01.contoso.com heartbeat: [11018]: info: Starting "/usr/lib64/heartbeat/ipfail" as uid 498  gid 499 (pid 11018)
Sep 27 05:56:53 heartbeat01.contoso.com heartbeat: [10992]: info: remote resource transition completed.
Sep 27 05:56:53 heartbeat01.contoso.com heartbeat: [10992]: info: remote resource transition completed.
Sep 27 05:56:53 heartbeat01.contoso.com heartbeat: [10992]: info: Local Resource acquisition completed. (none)
Sep 27 05:56:54 heartbeat01.contoso.com heartbeat: [10992]: info: heartbeat02.contoso.com wants to go standby [foreign]
Sep 27 05:56:59 heartbeat01.contoso.com heartbeat: [10992]: info: standby: acquire [foreign] resources from heartbeat02.contoso.com
Sep 27 05:56:59 heartbeat01.contoso.com heartbeat: [11038]: info: acquire local HA resources (standby).
ResourceManager(default)[11051]: 2016/09/27_05:56:59 info: Acquiring resource group: heartbeat01.contoso.com IPaddr::192.168.49.100/24/eth1 drbddisk::test Filesystem::/dev/drbd0::/data::ext4 mysqld
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100)[11079]: 2016/09/27_05:56:59 INFO:  Resource is stopped
ResourceManager(default)[11051]: 2016/09/27_05:56:59 info: Running /etc/ha.d/resource.d/IPaddr 192.168.49.100/24/eth1 start
IPaddr(IPaddr_192.168.49.100)[11201]: 2016/09/27_05:56:59 INFO: Adding inet address 192.168.49.100/24 to device eth1
IPaddr(IPaddr_192.168.49.100)[11201]: 2016/09/27_05:56:59 INFO: Bringing device eth1 up
IPaddr(IPaddr_192.168.49.100)[11201]: 2016/09/27_05:56:59 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.49.100 eth1 192.168.49.100 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.49.100)[11175]: 2016/09/27_05:56:59 INFO:  Success
ResourceManager(default)[11051]: 2016/09/27_05:56:59 info: Running /etc/ha.d/resource.d/drbddisk test start
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[11330]: 2016/09/27_05:56:59 INFO:  Resource is stopped
ResourceManager(default)[11051]: 2016/09/27_05:56:59 info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /data ext4 start
Filesystem(Filesystem_/dev/drbd0)[11412]: 2016/09/27_05:56:59 INFO: Running start for /dev/drbd0 on /data
/usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd0)[11404]: 2016/09/27_05:56:59 INFO:  Success
ResourceManager(default)[11051]: 2016/09/27_05:56:59 info: Running /etc/init.d/mysqld  start
Sep 27 05:57:00 heartbeat01.contoso.com ipfail: [11018]: info: Ping node count is balanced.
Sep 27 05:57:00 heartbeat01.contoso.com heartbeat: [11038]: info: local HA resource acquisition completed (standby).
Sep 27 05:57:00 heartbeat01.contoso.com heartbeat: [10992]: info: Standby resource acquisition done [foreign].
Sep 27 05:57:00 heartbeat01.contoso.com heartbeat: [10992]: info: Initial resource acquisition complete (auto_failback)
Sep 27 05:57:01 heartbeat01.contoso.com heartbeat: [10992]: info: remote resource transition completed.

账号		自动登录	找回密码
密码			立即注册

VMware vcenter+vSphere 6.5 U2共享

【跟谁学】韩宇极简英语课-技术人员不得不

用Zabbix通过JMX方式监控weblogic

winhex数据恢复教程（非常巨大，内容丰富）

Symantec Backup Exec 2015 2016/2012 BE20

NetScaler VPX部署之：NetScaler Gateway调

zabbix3.4.1安装部署+微信推送信息+大屏显

[经验分享] Heartbeat+Drbd+MySQL高可用

扫码加入运维网微信交流群