pursuer.chen(陈敏华)学而时习之

阿娇开发978

Fri Aug 26 11:57:36 2016 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)　　
Fri Aug 26 11:57:36 2016 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/mysql/log --output_file=/tmp/save_binary_logs_test --manager_version=0.55 --binlog_prefix=mysql-bin
　　
Fri Aug 26 11:57:36 2016 - [info] Executing seconary network check script: /usr/local/mha/bin/masterha_secondary_check -s backup -s master --user=root --master_host=master --master_ip=192.168.137.10 --master_port=3306  --user=root  --master_host=192.168.137.20  --master_ip=192.168.137.20  --master_port=3306
　　
Fri Aug
26 11:57:37 2016 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.137.20' (111))　　
Fri Aug 26 11:57:37 2016 - [warning] Connection failed 1 time(s)..
　　
Fri Aug 26 11:57:38 2016 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.137.20' (111))
　　
Fri Aug 26 11:57:38 2016 - [warning] Connection failed 2 time(s)..
　　
Fri Aug 26 11:57:38 2016 - [info] HealthCheck: SSH to 192.168.137.20 is reachable.
　　
Monitoring server backup is reachable, Master is not reachable from backup. OK.
　　
Fri Aug 26 11:57:39 2016 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.137.20' (111))
　　
Fri Aug 26 11:57:39 2016 - [warning] Connection failed 3 time(s)..
　　
Monitoring server master is reachable, Master is not reachable from master. OK.
　　
Fri Aug 26 11:57:41 2016 - [info] Master is not reachable from all other monitoring servers. Failover should start.
　　
Fri Aug 26 11:57:41 2016 - [warning] Master is not reachable from health checker!
　　
Fri Aug 26 11:57:41 2016 - [warning] Master 192.168.137.20(192.168.137.20:3306) is not reachable!
　　
Fri Aug 26 11:57:41 2016 - [warning] SSH is reachable.
　　
Fri Aug 26 11:57:41 2016 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /usr/local/mha/ha1/ha1.cnf again, and trying to connect to all servers to check server status..
　　
Fri Aug 26 11:57:41 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
　　
Fri Aug 26 11:57:41 2016 - [info] Reading application default configurations from /usr/local/mha/ha1/ha1.cnf..
　　
Fri Aug 26 11:57:41 2016 - [info] Reading server configurations from /usr/local/mha/ha1/ha1.cnf..
　　
Fri Aug 26 11:57:42 2016 - [info] Dead Servers:
　　
Fri Aug 26 11:57:42 2016 - [info]   192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:42 2016 - [info] Alive Servers:
　　
Fri Aug 26 11:57:42 2016 - [info]   192.168.137.10(192.168.137.10:3306)
　　
Fri Aug 26 11:57:42 2016 - [info]   192.168.137.30(192.168.137.30:3306)
　　
Fri Aug 26 11:57:42 2016 - [info] Alive Slaves:
　　
Fri Aug 26 11:57:42 2016 - [info]   192.168.137.10(192.168.137.10:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:42 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:42 2016 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Fri Aug 26 11:57:42 2016 - [info]   192.168.137.30(192.168.137.30:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:42 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:42 2016 - [info]     Not candidate for the new Master (no_master is set)
　　
Fri Aug 26 11:57:42 2016 - [info] Checking slave configurations..
　　
Fri Aug 26 11:57:42 2016 - [info]  read_only=1 is not set on slave 192.168.137.10(192.168.137.10:3306).
　　
Fri Aug 26 11:57:42 2016 - [info] Checking replication filtering settings..
　　
Fri Aug 26 11:57:42 2016 - [info]  Replication filtering check ok.
　　
Fri Aug 26 11:57:42 2016 - [info] Master is down!
　　
Fri Aug 26 11:57:42 2016 - [info] Terminating monitoring script.
　　
Fri Aug 26 11:57:42 2016 - [info] Got exit code 20 (Master dead).
　　
Fri Aug 26 11:57:42 2016 - [info] MHA::MasterFailover version 0.55.
　　
Fri Aug 26 11:57:42 2016 - [info] Starting master failover.
　　
Fri Aug 26 11:57:42 2016 - [info]
　　
Fri Aug 26 11:57:42 2016 - [info] * Phase 1: Configuration Check Phase..
　　
Fri Aug 26 11:57:42 2016 - [info]
　　
Fri Aug 26 11:57:44 2016 - [info] Dead Servers:
　　
Fri Aug 26 11:57:44 2016 - [info]   192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:44 2016 - [info] Checking master reachability via mysql(double check)..
　　
Fri Aug 26 11:57:44 2016 - [info]  ok.
　　
Fri Aug 26 11:57:44 2016 - [info] Alive Servers:
　　
Fri Aug 26 11:57:44 2016 - [info]   192.168.137.10(192.168.137.10:3306)
　　
Fri Aug 26 11:57:44 2016 - [info]   192.168.137.30(192.168.137.30:3306)
　　
Fri Aug 26 11:57:44 2016 - [info] Alive Slaves:
　　
Fri Aug 26 11:57:44 2016 - [info]   192.168.137.10(192.168.137.10:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:44 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:44 2016 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Fri Aug 26 11:57:44 2016 - [info]   192.168.137.30(192.168.137.30:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:44 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:44 2016 - [info]     Not candidate for the new Master (no_master is set)
　　
Fri Aug 26 11:57:44 2016 - [info] ** Phase 1: Configuration Check Phase completed.
　　
Fri Aug 26 11:57:44 2016 - [info]
　　
Fri Aug 26 11:57:44 2016 - [info] * Phase 2: Dead Master Shutdown Phase..
　　
Fri Aug 26 11:57:44 2016 - [info]
　　
Fri Aug 26 11:57:44 2016 - [info] Forcing shutdown so that applications never connect to the current master..
　　
Fri Aug 26 11:57:44 2016 - [info] Executing master IP deactivatation script:
　　
Fri Aug 26 11:57:44 2016 - [info]   /usr/local/mha/ha1/fail_script/master_ip_failover --orig_master_host=192.168.137.20 --orig_master_ip=192.168.137.20 --orig_master_port=3306 --command=stopssh --ssh_user=root  
　　

　　

　　
IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.137.50/24===
　　

　　
Disabling the VIP on old master: 192.168.137.20
　　
Fri Aug 26 11:57:45 2016 - [info]  done.
　　
Fri Aug 26 11:57:45 2016 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
　　
Fri Aug 26 11:57:45 2016 - [info] * Phase 2: Dead Master Shutdown Phase completed.
　　
Fri Aug 26 11:57:45 2016 - [info]
　　
Fri Aug 26 11:57:45 2016 - [info] * Phase 3: Master Recovery Phase..
　　
Fri Aug 26 11:57:45 2016 - [info]
　　
Fri Aug 26 11:57:45 2016 - [info] * Phase 3.1: Getting Latest Slaves Phase..
　　
Fri Aug 26 11:57:45 2016 - [info]
　　
Fri Aug 26 11:57:45 2016 - [info] The latest binary log file/position on all slaves is mysql-bin.000074:22461852

　　
Fri Aug 26 11:57:45 2016 - [info] Latest slaves (Slaves that received>　　
Fri Aug 26 11:57:45 2016 - [info]   192.168.137.30(192.168.137.30:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:45 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:45 2016 - [info]     Not candidate for the new Master (no_master is set)
　　
Fri Aug 26 11:57:45 2016 - [info] The oldest binary log file/position on all slaves is mysql-bin.000074:9857376
　　
Fri Aug 26 11:57:45 2016 - [info] Oldest slaves:
　　
Fri Aug 26 11:57:45 2016 - [info]   192.168.137.10(192.168.137.10:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:45 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:45 2016 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Fri Aug 26 11:57:45 2016 - [info]
　　
Fri Aug 26 11:57:45 2016 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
　　
Fri Aug 26 11:57:45 2016 - [info]
　　
Fri Aug 26 11:57:46 2016 - [info] Fetching dead master's binary logs..
　　
Fri Aug 26 11:57:46 2016 - [info] Executing command on the dead master 192.168.137.20(192.168.137.20:3306): save_binary_logs --command=save --start_file=mysql-bin.000074  --start_pos=22461852 --binlog_dir=/mysql/log --output_file=/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.55
　　
Creating /tmp if not exists..    ok.
　　
Concat binary/relay logs from mysql-bin.000074 pos 22461852 to mysql-bin.000074 EOF into /tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog ..
　　
Dumping binlog format description event, from position 0 to 120.. ok.
　　
Dumping effective binlog data from /mysql/log/mysql-bin.000074 position 22461852 to tail(22497564).. ok.
　　
Concat succeeded.
　　
Fri Aug 26 11:57:49 2016 - [info] scp from root@192.168.137.20:/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog to local:/usr/local/mha/ha1/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog succeeded.
　　
Fri Aug 26 11:57:52 2016 - [info] HealthCheck: SSH to 192.168.137.10 is reachable.
　　
Fri Aug 26 11:57:55 2016 - [info] HealthCheck: SSH to 192.168.137.30 is reachable.
　　
Fri Aug 26 11:57:55 2016 - [info]
　　
Fri Aug 26 11:57:55 2016 - [info] * Phase 3.3: Determining New Master Phase..
　　
Fri Aug 26 11:57:55 2016 - [info]

　　
Fri Aug 26 11:57:55 2016 - [info] Finding the latest slave that has all>
　　
Fri Aug 26 11:57:55 2016 - [info] Checking whether 192.168.137.30 has>　　
Fri Aug 26 11:57:55 2016 - [info] Executing command: apply_diff_relay_logs --command=find --latest_mlf=mysql-bin.000074 --latest_rmlp=22461852 --target_mlf=mysql-bin.000074 --target_rmlp=9857376 --server_id=30 --workdir=/tmp --timestamp=20160826115742 --manager_version=0.55 --relay_log_info=/mysql/data/relay-log.info  --relay_dir=/mysql/data/  :
　　
Opening /mysql/data/relay-log.info ... ok.
　　
Relay log found at /mysql/data, up to mysql-relay-bin.000003

　　
Fast>
　　
Target>
　　
Target>
　　
Fri Aug 26 11:57:56 2016 - [info] OK. 192.168.137.30 has all>　　
Fri Aug 26 11:57:56 2016 - [info] Searching new master from slaves..
　　
Fri Aug 26 11:57:56 2016 - [info]  Candidate masters from the configuration file:
　　
Fri Aug 26 11:57:56 2016 - [info]   192.168.137.10(192.168.137.10:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:56 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:56 2016 - [info]     Primary candidate for the new Master (candidate_master is set)
　　
Fri Aug 26 11:57:56 2016 - [info]  Non-candidate masters:
　　
Fri Aug 26 11:57:56 2016 - [info]   192.168.137.30(192.168.137.30:3306)  Version=5.6.15-log (oldest major version between slaves) log-bin:enabled
　　
Fri Aug 26 11:57:56 2016 - [info]     Replicating from 192.168.137.20(192.168.137.20:3306)
　　
Fri Aug 26 11:57:56 2016 - [info]     Not candidate for the new Master (no_master is set)

　　
Fri Aug 26 11:57:56 2016 - [info]  Searching from candidate_master slaves which have received the latest>　　
Fri Aug 26 11:57:56 2016 - [info]   Not found.
　　
Fri Aug 26 11:57:56 2016 - [info]  Searching from all candidate_master slaves..
　　
Fri Aug 26 11:57:56 2016 - [info] New master is 192.168.137.10(192.168.137.10:3306)
　　
Fri Aug 26 11:57:56 2016 - [info] Starting master failover..
　　
Fri Aug 26 11:57:56 2016 - [info]
　　
From:
　　
192.168.137.20 (current master)
　　
+--192.168.137.10
　　
+--192.168.137.30
　　

　　
To:
　　
192.168.137.10 (new master)
　　
+--192.168.137.30
　　
Fri Aug 26 11:57:56 2016 - [info]
　　
Fri Aug 26 11:57:56 2016 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
　　
Fri Aug 26 11:57:56 2016 - [info]

　　
Fri Aug 26 11:57:56 2016 - [info] Server 192.168.137.10 received>
　　
Fri Aug 26 11:57:56 2016 - [info] Need to get diffs from the latest slave(192.168.137.30) up to: mysql-bin.000074:22461852 (using the latest slave's>
　　
Fri Aug 26 11:57:56 2016 - [info] Connecting to the latest slave host 192.168.137.30, generating diff>　　
Fri Aug 26 11:57:56 2016 - [info] Executing command: apply_diff_relay_logs --command=generate_and_send --scp_user=root --scp_host=192.168.137.10 --latest_mlf=mysql-bin.000074 --latest_rmlp=22461852 --target_mlf=mysql-bin.000074 --target_rmlp=9857376 --server_id=30 --diff_file_readtolatest=/tmp/relay_from_read_to_latest_192.168.137.10_3306_20160826115742.binlog --workdir=/tmp --timestamp=20160826115742 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.55 --relay_log_info=/mysql/data/relay-log.info  --relay_dir=/mysql/data/
　　
Fri Aug 26 11:58:02 2016 - [info]
　　
Opening /mysql/data/relay-log.info ... ok.
　　
Relay log found at /mysql/data, up to mysql-relay-bin.000003

　　
Fast>
　　
Target>　　
Concat binary/relay logs from mysql-relay-bin.000003 pos 9857539 to mysql-relay-bin.000003 EOF into /tmp/relay_from_read_to_latest_192.168.137.10_3306_20160826115742.binlog ..
　　
Dumping binlog format description event, from position 0 to 283.. ok.
　　
Dumping effective binlog data from /mysql/data/mysql-relay-bin.000003 position 9857539 to tail(22462015).. ok.
　　
Concat succeeded.

　　
Generating diff>　　
scp slave:/tmp/relay_from_read_to_latest_192.168.137.10_3306_20160826115742.binlog to root@192.168.137.10(22) succeeded.
　　
Fri Aug 26 11:58:02 2016 - [info]  Generating diff files succeeded.
　　
Fri Aug 26 11:58:02 2016 - [info] Sending binlog..
　　
Fri Aug 26 11:58:04 2016 - [info] scp from local:/usr/local/mha/ha1/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog to root@192.168.137.10:/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog succeeded.
　　
Fri Aug 26 11:58:04 2016 - [info]
　　
Fri Aug 26 11:58:04 2016 - [info] * Phase 3.4: Master Log Apply Phase..
　　
Fri Aug 26 11:58:04 2016 - [info]
　　
Fri Aug 26 11:58:04 2016 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
　　
Fri Aug 26 11:58:04 2016 - [info] Starting recovery on 192.168.137.10(192.168.137.10:3306)..
　　
Fri Aug 26 11:58:04 2016 - [info]  Generating diffs succeeded.

　　
Fri Aug 26 11:58:04 2016 - [info] Waiting until all>　　
Fri Aug 26 12:00:06 2016 - [info]  done.
　　
Fri Aug 26 12:00:06 2016 - [info] Getting slave status..
　　
Fri Aug 26 12:00:06 2016 - [info] This slave(192.168.137.10)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000074:9857376). No need to recover from Exec_Master_Log_Pos.
　　
Fri Aug 26 12:00:06 2016 - [info] Connecting to the target slave host 192.168.137.10, running recover script..
　　
Fri Aug 26 12:00:06 2016 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=192.168.137.10 --slave_ip=192.168.137.10  --slave_port=3306 --apply_files=/tmp/relay_from_read_to_latest_192.168.137.10_3306_20160826115742.binlog,/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog --workdir=/tmp --target_version=5.6.15-log --timestamp=20160826115742 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.55 --slave_pass=xxx
　　
Fri Aug 26 12:04:22 2016 - [info]
　　
Concat all apply files to /tmp/total_binlog_for_192.168.137.10_3306.20160826115742.binlog ..
　　
Copying the first binlog file /tmp/relay_from_read_to_latest_192.168.137.10_3306_20160826115742.binlog to /tmp/total_binlog_for_192.168.137.10_3306.20160826115742.binlog.. ok.
　　
Dumping binlog head events (rotate events), skipping format description events from /tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog.. dumped up to pos 120. ok.
　　
/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog has effective binlog events from pos 120.
　　
Dumping effective binlog data from /tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog position 120 to tail(35832).. ok.
　　
Concat succeeded.
　　
All apply target binary logs are concatinated at /tmp/total_binlog_for_192.168.137.10_3306.20160826115742.binlog .
　　
MySQL client version is 5.6.15. Using --binary-mode.
　　
Applying differential binary/relay log files /tmp/relay_from_read_to_latest_192.168.137.10_3306_20160826115742.binlog,/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog on 192.168.137.10:3306. This may take long time...
　　
Applying log files succeeded.

　　
Fri Aug 26 12:04:22 2016 - [info]  All>　　
Fri Aug 26 12:04:22 2016 - [info] Getting new master's binlog name and position..
　　
Fri Aug 26 12:04:22 2016 - [info]  mysql-bin.000143:22123166
　　
Fri Aug 26 12:04:22 2016 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.137.10', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000143', MASTER_LOG_POS=22123166, MASTER_USER='repl', MASTER_PASSWORD='xxx';
　　
Fri Aug 26 12:04:22 2016 - [info] Executing master IP activate script:
　　
Fri Aug 26 12:04:22 2016 - [info]   /usr/local/mha/ha1/fail_script/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.137.20 --orig_master_ip=192.168.137.20 --orig_master_port=3306 --new_master_host=192.168.137.10 --new_master_ip=192.168.137.10 --new_master_port=3306 --new_master_user='root' --new_master_password='root'  
　　

　　

　　
IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.137.50/24===
　　

　　
Enabling the VIP - 192.168.137.50/24 on the new master - 192.168.137.10
　　
Fri Aug 26 12:04:25 2016 - [info]  OK.
　　
Fri Aug 26 12:04:25 2016 - [info] ** Finished master recovery successfully.
　　
Fri Aug 26 12:04:25 2016 - [info] * Phase 3: Master Recovery Phase completed.
　　
Fri Aug 26 12:04:25 2016 - [info]
　　
Fri Aug 26 12:04:25 2016 - [info] * Phase 4: Slaves Recovery Phase..
　　
Fri Aug 26 12:04:25 2016 - [info]
　　
Fri Aug 26 12:04:25 2016 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
　　
Fri Aug 26 12:04:25 2016 - [info]
　　
Fri Aug 26 12:04:25 2016 - [info] -- Slave diff file generation on host 192.168.137.30(192.168.137.30:3306) started, pid: 5029. Check tmp log /usr/local/mha/ha1/192.168.137.30_3306_20160826115742.log if it takes time..
　　
Fri Aug 26 12:04:26 2016 - [info]
　　
Fri Aug 26 12:04:26 2016 - [info] Log messages from 192.168.137.30 ...
　　
Fri Aug 26 12:04:26 2016 - [info]

　　
Fri Aug 26 12:04:25 2016 - [info]  This server has all>　　
Fri Aug 26 12:04:26 2016 - [info] End of log messages from 192.168.137.30.

　　
Fri Aug 26 12:04:26 2016 - [info] -- 192.168.137.30(192.168.137.30:3306) has the latest>
　　
Fri Aug 26 12:04:26 2016 - [info] Generating>　　
Fri Aug 26 12:04:26 2016 - [info]
　　
Fri Aug 26 12:04:26 2016 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
　　
Fri Aug 26 12:04:26 2016 - [info]
　　
Fri Aug 26 12:04:26 2016 - [info] -- Slave recovery on host 192.168.137.30(192.168.137.30:3306) started, pid: 5031. Check tmp log /usr/local/mha/ha1/192.168.137.30_3306_20160826115742.log if it takes time..
　　
Fri Aug 26 12:04:32 2016 - [info]
　　
Fri Aug 26 12:04:32 2016 - [info] Log messages from 192.168.137.30 ...
　　
Fri Aug 26 12:04:32 2016 - [info]
　　
Fri Aug 26 12:04:26 2016 - [info] Sending binlog..
　　
Fri Aug 26 12:04:28 2016 - [info] scp from local:/usr/local/mha/ha1/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog to root@192.168.137.30:/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog succeeded.
　　
Fri Aug 26 12:04:28 2016 - [info] Starting recovery on 192.168.137.30(192.168.137.30:3306)..
　　
Fri Aug 26 12:04:28 2016 - [info]  Generating diffs succeeded.

　　
Fri Aug 26 12:04:28 2016 - [info] Waiting until all>　　
Fri Aug 26 12:04:28 2016 - [info]  done.
　　
Fri Aug 26 12:04:28 2016 - [info] Getting slave status..
　　
Fri Aug 26 12:04:28 2016 - [info] This slave(192.168.137.30)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000074:22461852). No need to recover from Exec_Master_Log_Pos.
　　
Fri Aug 26 12:04:28 2016 - [info] Connecting to the target slave host 192.168.137.30, running recover script..
　　
Fri Aug 26 12:04:28 2016 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=192.168.137.30 --slave_ip=192.168.137.30  --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog --workdir=/tmp --target_version=5.6.15-log --timestamp=20160826115742 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.55 --slave_pass=xxx
　　
Fri Aug 26 12:04:30 2016 -[info]
　　
MySQL client version is 5.6.15. Using --binary-mode.
　　
Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.137.20_3306_20160826115742.binlog on 192.168.137.30:3306. This may take long time...
　　
Applying log files succeeded.

　　
Fri Aug 26 12:04:30 2016 - [info]  All>　　
Fri Aug 26 12:04:30 2016 - [info]  Resetting slave 192.168.137.30(192.168.137.30:3306) and starting replication from the new master 192.168.137.10(192.168.137.10:3306)..
　　
Fri Aug 26 12:04:31 2016 -[info]  Executed CHANGE MASTER.
　　
Fri Aug 26 12:04:31 2016 - [info]  Slave started.
　　
Fri Aug 26 12:04:32 2016 - [info] End of log messages from 192.168.137.30.
　　
Fri Aug 26 12:04:32 2016 - [info] -- Slave recovery on host 192.168.137.30(192.168.137.30:3306) succeeded.
　　
Fri Aug 26 12:04:32 2016 - [info] All new slave servers recovered successfully.
　　
Fri Aug 26 12:04:32 2016 - [info]
　　
Fri Aug 26 12:04:32 2016 - [info] * Phase 5: New master cleanup phase..
　　
Fri Aug 26 12:04:32 2016 - [info]
　　
Fri Aug 26 12:04:32 2016 - [info] Resetting slave info on the new master..
　　
Fri Aug 26 12:04:32 2016 - [info]  192.168.137.10: Resetting slave info succeeded.
　　
Fri Aug 26 12:04:32 2016 - [info] Master failover to 192.168.137.10(192.168.137.10:3306) completed successfully.
　　
Fri Aug 26 12:04:32 2016 - [info]
　　

　　
----- Failover Report -----
　　

　　
ha1: MySQL Master failover 192.168.137.20 to 192.168.137.10 succeeded
　　

　　
Master 192.168.137.20 is down!
　　

　　
Check MHA Manager logs at monitor:/usr/local/mha/ha1/manager.log for details.
　　

　　
Started automated(non-interactive) failover.
　　
Invalidated master IP address on 192.168.137.20.

　　
The latest slave 192.168.137.30(192.168.137.30:3306) has all>　　
Selected 192.168.137.10 as a new master.
　　
192.168.137.10: OK: Applying all logs succeeded.
　　
192.168.137.10: OK: Activated master IP address.

　　
192.168.137.30: This host has the latest>
　　
Generating>　　
192.168.137.30: OK: Applying all logs succeeded. Slave started, replicating from 192.168.137.10.
　　
192.168.137.10: Resetting slave info succeeded.
　　
Master failover to 192.168.137.10(192.168.137.10:3306) completed successfully.