MySQL高可用MHA部署 （一主二从）
Mon Jan 18 18:05:49 2016 - Goterror on MySQL select ping: 2006 (MySQL server has gone away)Mon Jan 18 18:05:49 2016 - ExecutingSSH check script: save_binary_logs --command=test --start_pos=4--binlog_dir=/data/dbdata/mysqllog/binlog--output_file=/tmp/save_binary_logs_test --manager_version=0.56--binlog_prefix=binlog
Creating /tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /data/dbdata/mysqllog/binlog, up to binlog.000003
Mon Jan 18 18:05:50 2016 - HealthCheck: SSH to 192.168.6.85 is reachable.
Mon Jan 18 18:05:50 2016 - Goterror on MySQL connect: 2013 (Lost connection to MySQL server at 'readinginitial communication packet', system error: 111)
Mon Jan 18 18:05:50 2016 - Connection failed 2 time(s)..
Mon Jan 18 18:05:51 2016 - Goterror on MySQL connect: 2013 (Lost connection to MySQL server at 'readinginitial communication packet', system error: 111)
Mon Jan 18 18:05:51 2016 - Connection failed 3 time(s)..
Mon Jan 18 18:05:52 2016 - Goterror on MySQL connect: 2013 (Lost connection to MySQL server at 'readinginitial communication packet', system error: 111)
Mon Jan 18 18:05:52 2016 - Connection failed 4 time(s)..
Mon Jan 18 18:05:52 2016 - Masteris not reachable from health checker!
Mon Jan 18 18:05:52 2016 - Master192.168.6.85(192.168.6.85:3306) is not reachable!
Mon Jan 18 18:05:52 2016 - SSH isreachable.
Mon Jan 18 18:05:52 2016 - Connecting to a master server failed. Reading configuration file/etc/masterha_default.cnf and /etc/mha/masterha_default.cnf again, and tryingto connect to all servers to check server status..
Mon Jan 18 18:05:52 2016 - Globalconfiguration file /etc/masterha_default.cnf not found. Skipping.
Mon Jan 18 18:05:52 2016 - Readingapplication default configuration from /etc/mha/masterha_default.cnf..
Mon Jan 18 18:05:52 2016 - Readingserver configuration from /etc/mha/masterha_default.cnf..
Mon Jan 18 18:05:52 2016 - GTIDfailover mode = 0
Mon Jan 18 18:05:52 2016 - DeadServers:
Mon Jan 18 18:05:52 2016 - 192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:52 2016 - AliveServers:
Mon Jan 18 18:05:52 2016 - 192.168.6.91(192.168.6.91:3306)
Mon Jan 18 18:05:52 2016 - 192.168.6.149(192.168.6.149:3306)
Mon Jan 18 18:05:52 2016 - AliveSlaves:
Mon Jan 18 18:05:52 2016 - 192.168.6.91(192.168.6.91:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:52 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:52 2016 - Primary candidate for the new Master(candidate_master is set)
Mon Jan 18 18:05:52 2016 - 192.168.6.149(192.168.6.149:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:52 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:52 2016 - Checkingslave configurations..
Mon Jan 18 18:05:52 2016 - read_only=1 is not set on slave192.168.6.91(192.168.6.91:3306).
Mon Jan 18 18:05:52 2016 - relay_log_purge=0 is not set on slave192.168.6.91(192.168.6.91:3306).
Mon Jan 18 18:05:52 2016 - read_only=1 is not set on slave192.168.6.149(192.168.6.149:3306).
Mon Jan 18 18:05:52 2016 - relay_log_purge=0 is not set on slave192.168.6.149(192.168.6.149:3306).
Mon Jan 18 18:05:52 2016 - Checkingreplication filtering settings..
Mon Jan 18 18:05:52 2016 - Replication filtering check ok.
Mon Jan 18 18:05:52 2016 - Master isdown!
Mon Jan 18 18:05:52 2016 - Terminating monitoring script.
Mon Jan 18 18:05:52 2016 - Got exitcode 20 (Master dead).
Mon Jan 18 18:05:52 2016 - Globalconfiguration file /etc/masterha_default.cnf not found. Skipping.
Mon Jan 18 18:05:52 2016 - Readingapplication default configuration from /etc/mha/masterha_default.cnf..
Mon Jan 18 18:05:52 2016 - Readingserver configuration from /etc/mha/masterha_default.cnf..
Mon Jan 18 18:05:52 2016 - MHA::MasterFailover version 0.56.
Mon Jan 18 18:05:52 2016 - Startingmaster failover.
Mon Jan 18 18:05:52 2016 -
Mon Jan 18 18:05:52 2016 - * Phase1: Configuration Check Phase..
Mon Jan 18 18:05:52 2016 -
Mon Jan 18 18:05:53 2016 - GTIDfailover mode = 0
Mon Jan 18 18:05:53 2016 - DeadServers:
Mon Jan 18 18:05:53 2016 - 192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:53 2016 - Checkingmaster reachability via MySQL(double check)...
Mon Jan 18 18:05:53 2016 - ok.
Mon Jan 18 18:05:53 2016 - AliveServers:
Mon Jan 18 18:05:53 2016 - 192.168.6.91(192.168.6.91:3306)
Mon Jan 18 18:05:53 2016 - 192.168.6.149(192.168.6.149:3306)
Mon Jan 18 18:05:53 2016 - AliveSlaves:
Mon Jan 18 18:05:53 2016 - 192.168.6.91(192.168.6.91:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:53 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:53 2016 - Primary candidate for the new Master(candidate_master is set)
Mon Jan 18 18:05:53 2016 - 192.168.6.149(192.168.6.149:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:53 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:53 2016 - StartingNon-GTID based failover.
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - ** Phase1: Configuration Check Phase completed.
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - * Phase2: Dead Master Shutdown Phase..
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - Forcingshutdown so that applications never connect to the current master..
Mon Jan 18 18:05:53 2016 - master_ip_failover_script is not set. Skipping invalidating dead master IPaddress.
Mon Jan 18 18:05:53 2016 - shutdown_scriptis not set. Skipping explicit shutting down of the dead master.
Mon Jan 18 18:05:53 2016 - * Phase2: Dead Master Shutdown Phase completed.
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - * Phase3: Master Recovery Phase..
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - * Phase3.1: Getting Latest Slaves Phase..
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - Thelatest binary log file/position on all slaves is binlog.000003:107
Mon Jan 18 18:05:53 2016 - Latestslaves (Slaves that received relay log files to the latest):
Mon Jan 18 18:05:53 2016 - 192.168.6.91(192.168.6.91:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:53 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:53 2016 - Primary candidate for the new Master(candidate_master is set)
Mon Jan 18 18:05:53 2016 - 192.168.6.149(192.168.6.149:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:53 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:53 2016 - Theoldest binary log file/position on all slaves is binlog.000003:107
Mon Jan 18 18:05:53 2016 - Oldestslaves:
Mon Jan 18 18:05:53 2016 - 192.168.6.91(192.168.6.91:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:53 2016 - Replicating from 192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:53 2016 - Primary candidate for the new Master(candidate_master is set)
Mon Jan 18 18:05:53 2016 - 192.168.6.149(192.168.6.149:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:53 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - * Phase3.2: Saving Dead Master's Binlog Phase..
Mon Jan 18 18:05:53 2016 -
Mon Jan 18 18:05:53 2016 - Fetchingdead master's binary logs..
Mon Jan 18 18:05:53 2016 - Executingcommand on the dead master 192.168.6.85(192.168.6.85:3306): save_binary_logs--command=save --start_file=binlog.000003 --start_pos=107 --binlog_dir=/data/dbdata/mysqllog/binlog--output_file=/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlog--handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56
Creating /tmp if not exists.. ok.
Concat binary/relay logs from binlog.000003pos 107 to binlog.000003 EOF into/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlog ..
Dumping binlog format description event, from position 0 to 107.. ok.
Dumping effective binlog data from /data/dbdata/mysqllog/binlog/binlog.000003position 107 to tail(126).. ok.
Concat succeeded.
Mon Jan 18 18:05:54 2016 - scp fromroot@192.168.6.85:/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlogto local:/var/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlogsucceeded.
Mon Jan 18 18:05:54 2016 - HealthCheck: SSH to 192.168.6.91 is reachable.
Mon Jan 18 18:05:55 2016 - HealthCheck: SSH to 192.168.6.149 is reachable.
Mon Jan 18 18:05:55 2016 -
Mon Jan 18 18:05:55 2016 - * Phase3.3: Determining New Master Phase..
Mon Jan 18 18:05:55 2016 -
Mon Jan 18 18:05:55 2016 - Findingthe latest slave that has all relay logs for recovering other slaves..
Mon Jan 18 18:05:55 2016 - Allslaves received relay logs to the same position. No need to resync each other.
Mon Jan 18 18:05:55 2016 - Searchingnew master from slaves..
Mon Jan 18 18:05:55 2016 - Candidate masters from the configurationfile:
Mon Jan 18 18:05:55 2016 - 192.168.6.91(192.168.6.91:3306)Version=5.5.33-log (oldest major versionbetween slaves) log-bin:enabled
Mon Jan 18 18:05:55 2016 - Replicating from192.168.6.85(192.168.6.85:3306)
Mon Jan 18 18:05:55 2016 - Primary candidate for the new Master(candidate_master is set)
Mon Jan 18 18:05:55 2016 - Non-candidate masters:
Mon Jan 18 18:05:55 2016 - Searching from candidate_master slaves whichhave received the latest relay log events..
Mon Jan 18 18:05:55 2016 - New masteris 192.168.6.91(192.168.6.91:3306)
Mon Jan 18 18:05:55 2016 - Startingmaster failover..
Mon Jan 18 18:05:55 2016 -
From:
192.168.6.85(192.168.6.85:3306) (currentmaster)
+--192.168.6.91(192.168.6.91:3306)
+--192.168.6.149(192.168.6.149:3306)
To:
192.168.6.91(192.168.6.91:3306) (newmaster)
+--192.168.6.149(192.168.6.149:3306)
Mon Jan 18 18:05:55 2016 -
Mon Jan 18 18:05:55 2016 - * Phase3.3: New Master Diff Log Generation Phase..
Mon Jan 18 18:05:55 2016 -
Mon Jan 18 18:05:55 2016 - This server has all relay logs. No need togenerate diff files from the latest slave.
Mon Jan 18 18:05:55 2016 - Sendingbinlog..
Mon Jan 18 18:05:55 2016 - scp fromlocal:/var/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlogtoroot@192.168.6.91:/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlogsucceeded.
Mon Jan 18 18:05:55 2016 -
Mon Jan 18 18:05:55 2016 - * Phase3.4: Master Log Apply Phase..
Mon Jan 18 18:05:55 2016 -
Mon Jan 18 18:05:55 2016 - *NOTICE:If any error happens from this phase, manual recovery is needed.
Mon Jan 18 18:05:55 2016 - Startingrecovery on 192.168.6.91(192.168.6.91:3306)..
Mon Jan 18 18:05:55 2016 - Generating diffs succeeded.
Mon Jan 18 18:05:55 2016 - Waitinguntil all relay logs are applied.
Mon Jan 18 18:05:55 2016 - done.
Mon Jan 18 18:05:55 2016 - Gettingslave status..
Mon Jan 18 18:05:55 2016 - Thisslave(192.168.6.91)'s Exec_Master_Log_Pos equals toRead_Master_Log_Pos(binlog.000003:107). No need to recover fromExec_Master_Log_Pos.
Mon Jan 18 18:05:55 2016 - Connecting to the target slave host 192.168.6.91, running recover script..
Mon Jan 18 18:05:55 2016 - Executingcommand: apply_diff_relay_logs --command=apply --slave_user='root'--slave_host=192.168.6.91 --slave_ip=192.168.6.91--slave_port=3306--apply_files=/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlog--workdir=/tmp --target_version=5.5.33-log --timestamp=20160118180552--handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56--slave_pass=xxx
Mon Jan 18 18:05:56 2016 -
Applying differential binary/relay logfiles /tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlog on192.168.6.91:3306. This may take long time...
Applying log files succeeded.
Mon Jan 18 18:05:56 2016 - All relay logs were successfully applied.
Mon Jan 18 18:05:56 2016 - Gettingnew master's binlog name and position..
Mon Jan 18 18:05:56 2016 - binlog.000003:107
Mon Jan 18 18:05:56 2016 - All other slaves should start replicationfrom here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.6.91',MASTER_PORT=3306, MASTER_LOG_FILE='binlog.000003', MASTER_LOG_POS=107,MASTER_USER='repl', MASTER_PASSWORD='xxx';
Mon Jan 18 18:05:56 2016 - master_ip_failover_script is not set. Skipping taking over new master IPaddress.
Mon Jan 18 18:05:56 2016 - **Finished master recovery successfully.
Mon Jan 18 18:05:56 2016 - * Phase3: Master Recovery Phase completed.
Mon Jan 18 18:05:56 2016 -
Mon Jan 18 18:05:56 2016 - * Phase4: Slaves Recovery Phase..
Mon Jan 18 18:05:56 2016 -
Mon Jan 18 18:05:56 2016 - * Phase4.1: Starting Parallel Slave Diff Log Generation Phase..
Mon Jan 18 18:05:56 2016 -
Mon Jan 18 18:05:56 2016 - -- Slavediff file generation on host 192.168.6.149(192.168.6.149:3306) started, pid:32132. Check tmp log /var/tmp/192.168.6.149_3306_20160118180552.log if it takestime..
Mon Jan 18 18:05:56 2016 -
Mon Jan 18 18:05:56 2016 - Logmessages from 192.168.6.149 ...
Mon Jan 18 18:05:56 2016 -
Mon Jan 18 18:05:56 2016 - This server has all relay logs. No need togenerate diff files from the latest slave.
Mon Jan 18 18:05:56 2016 - End oflog messages from 192.168.6.149.
Mon Jan 18 18:05:56 2016 - --192.168.6.149(192.168.6.149:3306) has the latest relay log events.
Mon Jan 18 18:05:56 2016 - Generating relay diff files from the latest slave succeeded.
Mon Jan 18 18:05:56 2016 -
Mon Jan 18 18:05:56 2016 - * Phase4.2: Starting Parallel Slave Log Apply Phase..
Mon Jan 18 18:05:56 2016 -
Mon Jan 18 18:05:56 2016 - -- Slaverecovery on host 192.168.6.149(192.168.6.149:3306) started, pid: 32134. Checktmp log /var/tmp/192.168.6.149_3306_20160118180552.log if it takes time..
Mon Jan 18 18:05:57 2016 -
Mon Jan 18 18:05:57 2016 - Log messagesfrom 192.168.6.149 ...
Mon Jan 18 18:05:57 2016 -
Mon Jan 18 18:05:56 2016 - Sendingbinlog..
Mon Jan 18 18:05:56 2016 - scp fromlocal:/var/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlogto root@192.168.6.149:/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlogsucceeded.
Mon Jan 18 18:05:56 2016 - Startingrecovery on 192.168.6.149(192.168.6.149:3306)..
Mon Jan 18 18:05:56 2016 - Generating diffs succeeded.
Mon Jan 18 18:05:56 2016 - Waitinguntil all relay logs are applied.
Mon Jan 18 18:05:56 2016 - done.
Mon Jan 18 18:05:56 2016 - Gettingslave status..
Mon Jan 18 18:05:56 2016 - Thisslave(192.168.6.149)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(binlog.000003:107).No need to recover from Exec_Master_Log_Pos.
Mon Jan 18 18:05:56 2016 - Connecting to the target slave host 192.168.6.149, running recover script..
Mon Jan 18 18:05:56 2016 - Executingcommand: apply_diff_relay_logs --command=apply --slave_user='root'--slave_host=192.168.6.149 --slave_ip=192.168.6.149--slave_port=3306--apply_files=/tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlog--workdir=/tmp --target_version=5.5.33-log --timestamp=20160118180552--handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56--slave_pass=xxx
Mon Jan 18 18:05:57 2016 -
Applying differential binary/relay logfiles /tmp/saved_master_binlog_from_192.168.6.85_3306_20160118180552.binlog on192.168.6.149:3306. This may take long time...
Applying log files succeeded.
Mon Jan 18 18:05:57 2016 - All relay logs were successfully applied.
Mon Jan 18 18:05:57 2016 - Resetting slave 192.168.6.149(192.168.6.149:3306)and starting replication from the new master 192.168.6.91(192.168.6.91:3306)..
Mon Jan 18 18:05:57 2016 - Executed CHANGE MASTER.
Mon Jan 18 18:05:57 2016 - Slave started.
Mon Jan 18 18:05:57 2016 - End oflog messages from 192.168.6.149.
Mon Jan 18 18:05:57 2016 - -- Slaverecovery on host 192.168.6.149(192.168.6.149:3306) succeeded.
Mon Jan 18 18:05:57 2016 - All newslave servers recovered successfully.
Mon Jan 18 18:05:57 2016 -
Mon Jan 18 18:05:57 2016 - * Phase5: New master cleanup phase..
Mon Jan 18 18:05:57 2016 -
Mon Jan 18 18:05:57 2016 - Resettingslave info on the new master..
Mon Jan 18 18:05:57 2016 - 192.168.6.91: Resetting slave info succeeded.
Mon Jan 18 18:05:57 2016 - Masterfailover to 192.168.6.91(192.168.6.91:3306) completed successfully.
Mon Jan 18 18:05:57 2016 - Deletedserver1 entry from /etc/mha/masterha_default.cnf .
Mon Jan 18 18:05:57 2016 -
----- Failover Report -----
masterha_default: MySQL Master failover192.168.6.85(192.168.6.85:3306) to 192.168.6.91(192.168.6.91:3306) succeeded
Master 192.168.6.85(192.168.6.85:3306) isdown!
Check MHA Manager logs at slave2 fordetails.
Started automated(non-interactive)failover.
The latest slave192.168.6.91(192.168.6.91:3306) has all relay logs for recovery.
Selected 192.168.6.91(192.168.6.91:3306) asa new master.
192.168.6.91(192.168.6.91:3306): OK:Applying all logs succeeded.
192.168.6.149(192.168.6.149:3306): Thishost has the latest relay log events.
Generating relay diff files from the latestslave succeeded.
192.168.6.149(192.168.6.149:3306): OK:Applying all logs succeeded. Slave started, replicating from192.168.6.91(192.168.6.91:3306)
192.168.6.91(192.168.6.91:3306): Resettingslave info succeeded.
Master failover to192.168.6.91(192.168.6.91:3306) completed successfully.
页:
[1]