MySQL高可用之MHA

butai

关闭128主库　　shell> mysqladmin -u root -p shutdown
　　
　　查看切换日志
　　shell> tail -f /db/tool/masterha/app1/manager.log
　　
　　Fri Feb 13 01:19:26 2015 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)
　　Fri Feb 13 01:19:26 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4
　　--binlog_dir=/mysql/mysql/ --output_file=/var/tmp/save_binary_logs_test
　　--manager_version=0.55 --binlog_prefix=mysql-info
　　Fri Feb 13 01:19:26 2015 - [info] HealthCheck: SSH to 192.168.90.128 is reachable.
　　Fri Feb 13 01:19:29 2015 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.90.128')
　　Fri Feb 13 01:19:29 2015 - [warning] Connection failed 3 time(s)..
　　Fri Feb 13 01:19:29 2015 - [warning] Master is not reachable from health checker!
　　Fri Feb 13 01:19:29 2015 - [warning] Master 192.168.90.128(192.168.90.128:3306) is not reachable!
　　Fri Feb 13 01:19:29 2015 - [warning] SSH is reachable.
　　Fri Feb 13 01:19:29 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf..
　　Fri Feb 13 01:19:30 2015 - [info] Dead Servers:
　　Fri Feb 13 01:19:30 2015 - [info]   192.168.90.128(192.168.90.128:3306)
　　Fri Feb 13 01:19:30 2015 - [info] Alive Servers:
　　Fri Feb 13 01:19:30 2015 - [info]   192.168.90.129(192.168.90.129:3306)
　　Fri Feb 13 01:19:30 2015 - [info]   192.168.90.130(192.168.90.130:3306)
　　Fri Feb 13 01:19:30 2015 - [info]     Replicating from 192.168.90.128(192.168.90.128:3306)
　　Fri Feb 13 01:19:30 2015 - [info]     Not candidate for the new Master (no_master is set)
　　Fri Feb 13 01:19:30 2015 - [info] Checking slave configurations..
　　Fri Feb 13 01:19:30 2015 - [info] Checking replication filtering settings..
　　Fri Feb 13 01:19:30 2015 - [info]  Replication filtering check ok.
　　Fri Feb 13 01:19:30 2015 - [info] Master is down!
　　Fri Feb 13 01:19:30 2015 - [info] Terminating monitoring script.
　　Fri Feb 13 01:19:30 2015 - [info] Got exit code 20 (Master dead).
　　Fri Feb 13 01:19:30 2015 - [info] MHA::MasterFailover version 0.55.
　　Fri Feb 13 01:19:30 2015 - [info] Starting master failover.
　　Fri Feb 13 01:19:30 2015 - [info]
　　Fri Feb 13 01:19:30 2015 - [info] * Phase 1: Configuration Check Phase..
　　Fri Feb 13 01:19:31 2015 - [info] ** Phase 1: Configuration Check Phase completed.
　　Fri Feb 13 01:19:31 2015 - [info]
　　Fri Feb 13 01:19:31 2015 - [info] * Phase 2: Dead Master Shutdown Phase..
　　Fri Feb 13 01:19:32 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.
　　Fri Feb 13 01:19:32 2015 - [info]
　　Fri Feb 13 01:19:32 2015 - [info] * Phase 3: Master Recovery Phase..
　　Fri Feb 13 01:19:32 2015 - [info]
　　Fri Feb 13 01:19:32 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..
　　Fri Feb 13 01:19:32 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
　　Fri Feb 13 01:19:33 2015 - [info] * Phase 3.3: Determining New Master Phase..
　　
　　Fri Feb 13 01:19:33 2015 - [info] New master is 192.168.90.129(192.168.90.129:3306)
　　Fri Feb 13 01:19:33 2015 - [info] Starting master failover..
　　Fri Feb 13 01:19:33 2015 - [info]
　　From:
　　192.168.90.128 (current master)
　　+--192.168.90.129
　　+--192.168.90.130
　　
　　To:
　　192.168.90.129 (new master)
　　+--192.168.90.130
　　Fri Feb 13 01:19:33 2015 - [info]
　　Fri Feb 13 01:19:33 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
　　Fri Feb 13 01:19:33 2015 - [info] * Phase 3.4: Master Log Apply Phase..
　　Fri Feb 13 01:19:33 2015 - [info] Setting read_only=0 on 192.168.90.129(192.168.90.129:3306)..
　　Fri Feb 13 01:19:33 2015 - [info]  ok.
　　Fri Feb 13 01:19:33 2015 - [info] ** Finished master recovery successfully.
　　Fri Feb 13 01:19:33 2015 - [info] * Phase 3: Master Recovery Phase completed.
　　Fri Feb 13 01:19:33 2015 - [info]
　　Fri Feb 13 01:19:33 2015 - [info] * Phase 4: Slaves Recovery Phase..
　　Fri Feb 13 01:19:33 2015 - [info]
　　Fri Feb 13 01:19:33 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
　　Fri Feb 13 01:19:34 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
　　Fri Feb 13 01:19:34 2015 - [info]  Resetting slave 192.168.90.130(192.168.90.130:3306) and starting replication
　　from the new master 192.168.90.129(192.168.90.129:3306)..
　　Fri Feb 13 01:19:34 2015 - [info]  Executed CHANGE MASTER.
　　Fri Feb 13 01:19:34 2015 - [info]  Slave started.
　　Fri Feb 13 01:19:35 2015 - [info] End of log messages from 192.168.90.130.
　　Fri Feb 13 01:19:35 2015 - [info] -- Slave recovery on host 192.168.90.130(192.168.90.130:3306) succeeded.
　　Fri Feb 13 01:19:35 2015 - [info] All new slave servers recovered successfully.
　　Fri Feb 13 01:19:35 2015 - [info]
　　Fri Feb 13 01:19:35 2015 - [info] * Phase 5: New master cleanup phase..
　　Fri Feb 13 01:19:35 2015 - [info]
　　Fri Feb 13 01:19:35 2015 - [info] Resetting slave info on the new master..
　　Fri Feb 13 01:19:35 2015 - [info]  192.168.90.129: Resetting slave info succeeded.
　　Fri Feb 13 01:19:35 2015 - [info] Master failover to 192.168.90.129(192.168.90.129:3306) completed successfully.
　　Fri Feb 13 01:19:35 2015 - [info]
　　
　　----- Failover Report -----
　　
　　app1: MySQL Master failover 192.168.90.128 to 192.168.90.129 succeeded
　　
　　Master 192.168.90.128 is down!
　　
　　Check MHA Manager logs at rac1:/db/tool/masterha/app1/manager.log for details.
　　
　　Started automated(non-interactive) failover.

　　The latest slave 192.168.90.129(192.168.90.129:3306) has all>　　Selected 192.168.90.129 as a new master.
　　192.168.90.129: OK: Applying all logs succeeded.

　　192.168.90.130: This host has the latest>
　　Generating>　　192.168.90.130: OK: Applying all logs succeeded. Slave started, replicating from 192.168.90.129.
　　192.168.90.129: Resetting slave info succeeded.
　　Master failover to 192.168.90.129(192.168.90.129:3306) completed successfully.
　　
　　
　　重新启动128主库
　　shell> masterha_check_repl --conf=/etc/masterha/app1.cnf  
　　...
　　Tue Mar 17 14:04:36 2015 - [info]
　　192.168.90.129 (current master)
　　+--192.168.90.128
　　+--192.168.90.130
　　
　　Tue Mar 17 14:04:36 2015 - [info] Checking replication health on 192.168.90.128..
　　Tue Mar 17 14:04:36 2015 - [info]  ok.
　　Tue Mar 17 14:04:36 2015 - [info] Checking replication health on 192.168.90.130..
　　Tue Mar 17 14:04:36 2015 - [info]  ok.
　　Tue Mar 17 14:04:36 2015 - [warning] master_ip_failover_script is not defined.
　　Tue Mar 17 14:04:36 2015 - [warning] shutdown_script is not defined.
　　Tue Mar 17 14:04:36 2015 - [info] Got exit code 0 (Not master dead).
　　
　　MySQL Replication Health is OK.