MySql 的 MHA 配置

seemebaby

Tue Jul 14 16:43:59 2015 - [info] Set secondary check script: masterha_secondary_check -s dbsrv1 -s dbsrv3　　
Tue Jul 14 16:43:59 2015 - [info] Starting ping health check on dbsrv1(10.1.1.102:3306)..
　　
Tue Jul 14 16:43:59 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
　　
Tue Jul 14 16:44:29 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
　　
Tue Jul 14 16:44:29 2015 - [info] Executing secondary network check script: masterha_secondary_check -s dbsrv1 -s dbsrv3  --user=root  --master_host=dbsrv1  --master_ip=10.1.1.102  --master_port=3306 --master_user=repluser --master_password=replpass --ping_type=SELECT
　　
Tue Jul 14 16:44:29 2015 - [info] Executing SSH check script: exit 0
　　
Tue Jul 14 16:44:29 2015 - [info] HealthCheck: SSH to dbsrv1 is reachable.
　　
Monitoring server dbsrv1 is reachable, Master is not reachable from dbsrv1. OK.
　　
Monitoring server dbsrv3 is reachable, Master is not reachable from dbsrv3. OK.
　　
Tue Jul 14 16:44:29 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start.
　　
Tue Jul 14 16:44:32 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
　　
Tue Jul 14 16:44:32 2015 - [warning] Connection failed 2 time(s)..
　　
Tue Jul 14 16:44:35 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
　　
Tue Jul 14 16:44:35 2015 - [warning] Connection failed 3 time(s)..
　　
Tue Jul 14 16:44:38 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
　　
Tue Jul 14 16:44:38 2015 - [warning] Connection failed 4 time(s)..
　　
Tue Jul 14 16:44:38 2015 - [warning] Master is not reachable from health checker!
　　
Tue Jul 14 16:44:38 2015 - [warning] Master dbsrv1(10.1.1.102:3306) is not reachable!
　　
Tue Jul 14 16:44:38 2015 - [warning] SSH is reachable.
　　
Tue Jul 14 16:44:38 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1.cnf again, and trying to connect to all servers to check server status..
　　
Tue Jul 14 16:44:38 2015 - [info] Reading default configuration from /etc/masterha_default.cnf..
　　
Tue Jul 14 16:44:38 2015 - [info] Reading application default configuration from /etc/mha/app1.cnf..
　　
Tue Jul 14 16:44:38 2015 - [info] Reading server configuration from /etc/mha/app1.cnf..
　　
Tue Jul 14 16:44:38 2015 - [info] GTID failover mode = 1
　　
Tue Jul 14 16:44:38 2015 - [info] Dead Servers:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv1(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info] Alive Servers:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv2(10.1.1.107:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv3(10.1.1.108:3306)
　　
Tue Jul 14 16:44:38 2015 - [info] Alive Slaves:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv2(10.1.1.107:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Not candidate for the new Master (no_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv3(10.1.1.108:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info] Checking slave configurations..
　　
Tue Jul 14 16:44:38 2015 - [info]  read_only=1 is not set on slave dbsrv2(10.1.1.107:3306).
　　
Tue Jul 14 16:44:38 2015 - [info]  read_only=1 is not set on slave dbsrv3(10.1.1.108:3306).
　　
Tue Jul 14 16:44:38 2015 - [info] Checking replication filtering settings..
　　
Tue Jul 14 16:44:38 2015 - [info]  Replication filtering check ok.
　　
Tue Jul 14 16:44:38 2015 - [info] Master is down!
　　
Tue Jul 14 16:44:38 2015 - [info] Terminating monitoring script.
　　
Tue Jul 14 16:44:38 2015 - [info] Got exit code 20 (Master dead).
　　
Tue Jul 14 16:44:38 2015 - [info] MHA::MasterFailover version 0.56.
　　
Tue Jul 14 16:44:38 2015 - [info] Starting master failover.
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] * Phase 1: Configuration Check Phase..
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] GTID failover mode = 1
　　
Tue Jul 14 16:44:38 2015 - [info] Dead Servers:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv1(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info] Checking master reachability via MySQL(double check)...
　　
Tue Jul 14 16:44:38 2015 - [info]  ok.
　　
Tue Jul 14 16:44:38 2015 - [info] Alive Servers:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv2(10.1.1.107:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv3(10.1.1.108:3306)
　　
Tue Jul 14 16:44:38 2015 - [info] Alive Slaves:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv2(10.1.1.107:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Not candidate for the new Master (no_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv3(10.1.1.108:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info] Starting GTID based failover.
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] ** Phase 1: Configuration Check Phase completed.
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] * Phase 2: Dead Master Shutdown Phase..
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] Forcing shutdown so that applications never connect to the current master..
　　
Tue Jul 14 16:44:38 2015 - [info] Executing master IP deactivation script:
　　
Tue Jul 14 16:44:38 2015 - [info]   /etc/mha/master_ip_failover --orig_master_host=dbsrv1 --orig_master_ip=10.1.1.102 --orig_master_port=3306 --command=stopssh --ssh_user=root
　　

　　
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
　　

　　
Disabling the VIP on old master: dbsrv1
　　
Tue Jul 14 16:44:38 2015 - [info]  done.
　　
Tue Jul 14 16:44:38 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
　　
Tue Jul 14 16:44:38 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] * Phase 3: Master Recovery Phase..
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] The latest binary log file/position on all slaves is mysqlmaster-bin.000006:191
　　
Tue Jul 14 16:44:38 2015 - [info] Latest slaves (Slaves that received relay log files to the latest):
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv2(10.1.1.107:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Not candidate for the new Master (no_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv3(10.1.1.108:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info] The oldest binary log file/position on all slaves is mysqlmaster-bin.000006:191
　　
Tue Jul 14 16:44:38 2015 - [info] Oldest slaves:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv2(10.1.1.107:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Not candidate for the new Master (no_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv3(10.1.1.108:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] * Phase 3.3: Determining New Master Phase..
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] Searching new master from slaves..
　　
Tue Jul 14 16:44:38 2015 - [info]  Candidate masters from the configuration file:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv3(10.1.1.108:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info]  Non-candidate masters:
　　
Tue Jul 14 16:44:38 2015 - [info]   dbsrv2(10.1.1.107:3306)  Version=5.6.19-log (oldest major version between slaves) log-bin:enabled
　　
Tue Jul 14 16:44:38 2015 - [info]     GTID ON
　　
Tue Jul 14 16:44:38 2015 - [info]     Replicating from 10.1.1.102(10.1.1.102:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]     Not candidate for the new Master (no_master is set)
　　
Tue Jul 14 16:44:38 2015 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
　　
Tue Jul 14 16:44:38 2015 - [info] New master is dbsrv3(10.1.1.108:3306)
　　
Tue Jul 14 16:44:38 2015 - [info] Starting master failover..
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
From:
　　
dbsrv1(10.1.1.102:3306) (current master)
　　
+--dbsrv2(10.1.1.107:3306)
　　
+--dbsrv3(10.1.1.108:3306)
　　

　　
To:
　　
dbsrv3(10.1.1.108:3306) (new master)
　　
+--dbsrv2(10.1.1.107:3306)
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info] * Phase 3.3: New Master Recovery Phase..
　　
Tue Jul 14 16:44:38 2015 - [info]
　　
Tue Jul 14 16:44:38 2015 - [info]  Waiting all logs to be applied..
　　
Tue Jul 14 16:44:38 2015 - [info]   done.
　　
Tue Jul 14 16:44:38 2015 - [info]  Replicating from the latest slave dbsrv2(10.1.1.107:3306) and waiting to apply..
　　
Tue Jul 14 16:44:38 2015 - [info]  Waiting all logs to be applied on the latest slave..
　　
Tue Jul 14 16:44:38 2015 - [info]  Resetting slave dbsrv3(10.1.1.108:3306) and starting replication from the new master dbsrv2(10.1.1.107:3306)..
　　
Tue Jul 14 16:44:39 2015 - [info]  Executed CHANGE MASTER.
　　
Tue Jul 14 16:44:39 2015 - [info]  Slave started.
　　
Tue Jul 14 16:44:39 2015 - [info]  Waiting to execute all relay logs on dbsrv3(10.1.1.108:3306)..
　　
Tue Jul 14 16:44:39 2015 - [info]  master_pos_wait(mysqlslave-bin.000001:634) completed on dbsrv3(10.1.1.108:3306). Executed 3 events.
　　
Tue Jul 14 16:44:39 2015 - [info]   done.
　　
Tue Jul 14 16:44:39 2015 - [info]   done.
　　
Tue Jul 14 16:44:39 2015 - [info] Getting new master's binlog name and position..
　　
Tue Jul 14 16:44:39 2015 - [info]  mysqlslave-bin.000001:634
　　
Tue Jul 14 16:44:39 2015 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='dbsrv3 or 10.1.1.108', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repluser', MASTER_PASSWORD='xxx';
　　
Tue Jul 14 16:44:39 2015 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysqlslave-bin.000001, 634, 8c71815b-116f-11e4-b9e2-0050569f2c2d:1-2
　　
Tue Jul 14 16:44:39 2015 - [info] Executing master IP activate script:
　　
Tue Jul 14 16:44:39 2015 - [info]   /etc/mha/master_ip_failover --command=start --ssh_user=root --orig_master_host=dbsrv1 --orig_master_ip=10.1.1.102 --orig_master_port=3306 --new_master_host=dbsrv3 --new_master_ip=10.1.1.108 --new_master_port=3306 --new_master_user='repluser' --new_master_password='replpass'
　　

　　
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
　　

　　
Enabling the VIP - 10.1.1.110 on the new master - dbsrv3
　　
Tue Jul 14 16:44:40 2015 - [info]  OK.
　　
Tue Jul 14 16:44:40 2015 - [info] ** Finished master recovery successfully.
　　
Tue Jul 14 16:44:40 2015 - [info] * Phase 3: Master Recovery Phase completed.
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info] * Phase 4: Slaves Recovery Phase..
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info] * Phase 4.1: Starting Slaves in parallel..
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info] -- Slave recovery on host dbsrv2(10.1.1.107:3306) started, pid: 16698. Check tmp log /var/log/masterha/app1/dbsrv2_3306_20150714164438.log if it takes time..
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info] Log messages from dbsrv2 ...
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info]  Resetting slave dbsrv2(10.1.1.107:3306) and starting replication from the new master dbsrv3(10.1.1.108:3306)..
　　
Tue Jul 14 16:44:40 2015 - [info]  Executed CHANGE MASTER.
　　
Tue Jul 14 16:44:40 2015 - [info]  Slave started.
　　
Tue Jul 14 16:44:40 2015 - [info]  gtid_wait(8c71815b-116f-11e4-b9e2-0050569f2c2d:1-2) completed on dbsrv2(10.1.1.107:3306). Executed 0 events.
　　
Tue Jul 14 16:44:40 2015 - [info] End of log messages from dbsrv2.
　　
Tue Jul 14 16:44:40 2015 - [info] -- Slave on host dbsrv2(10.1.1.107:3306) started.
　　
Tue Jul 14 16:44:40 2015 - [info] All new slave servers recovered successfully.
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info] * Phase 5: New master cleanup phase..
　　
Tue Jul 14 16:44:40 2015 - [info]
　　
Tue Jul 14 16:44:40 2015 - [info] Resetting slave info on the new master..
　　
Tue Jul 14 16:44:41 2015 - [info]  dbsrv3: Resetting slave info succeeded.
　　
Tue Jul 14 16:44:41 2015 - [info] Master failover to dbsrv3(10.1.1.108:3306) completed successfully.
　　
Tue Jul 14 16:44:41 2015 - [info]
　　

　　
----- Failover Report -----
　　

　　
app1: MySQL Master failover dbsrv1(10.1.1.102:3306) to dbsrv3(10.1.1.108:3306) succeeded
　　

　　
Master dbsrv1(10.1.1.102:3306) is down!