mysql(innodb)故障

dfsfsf

收到短信报警，是一台R710硬盘(raid5 8块盘)坏了1块，联系机房更换硬盘，
因插槽问题 换了几块新的硬盘上去后还是亮黄灯，又将原来
1块硬盘换上后，亮绿灯，过了几分钟机器宕机了。。。。。
之后分析是主板问题，故又找了1台新的R710 将点不亮的机器上的
硬盘拔下换到新的R710上，将raid信息import后 卡在某个地方了，
发现硬盘有亮黄灯，再次更换坏的硬盘后，能进入系统了，进入系统后发现
mysql进程在，3306端口没有监听，mysql的error日志一直在输出

1、相关error日志
InnoDB: Log scan progressed past the checkpoint lsn 1613 2401579069
150613 20:34:29  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 1613 2406821888
InnoDB: Doing recovery: scanned up to log sequence number 1613 2412064768
InnoDB: Doing recovery: scanned up to log sequence number 1613 2417307648
InnoDB: Doing recovery: scanned up to log sequence number 1613 2422550528
InnoDB: Doing recovery: scanned up to log sequence number 1613 2427793408
InnoDB: Doing recovery: scanned up to log sequence number 1613 2433036288
InnoDB: Doing recovery: scanned up to log sequence number 1613 2438279168
InnoDB: Doing recovery: scanned up to log sequence number 1613 2443522048
InnoDB: Doing recovery: scanned up to log sequence number 1613 2448764928
InnoDB: Doing recovery: scanned up to log sequence number 1613 2454007808
InnoDB: Doing recovery: scanned up to log sequence number 1613 2459250688
InnoDB: Doing recovery: scanned up to log sequence number 1613 2464493568
InnoDB: Doing recovery: scanned up to log sequence number 1613 2469736448
InnoDB: Doing recovery: scanned up to log sequence number 1614 600860160
InnoDB: Doing recovery: scanned up to log sequence number 1614 606103040
InnoDB: Doing recovery: scanned up to log sequence number 1614 611345920
InnoDB: Doing recovery: scanned up to log sequence number 1614 613999763
150614  1:35:40  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 626966891, file name 81.000471
InnoDB: Last MySQL binlog file position 0 77012, file name /opt/mysql.bin/35.008407
150614  1:37:20  InnoDB: Started; log sequence number 1614 613999763
150614  1:37:20 [Note] Recovering after a crash using /opt/mysql.bin/35
150614  1:37:20 [Note] Starting crash recovery...
150614  1:37:20 [Note] Crash recovery finished.
150614  1:37:20 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=192-relay-bin' to avoid this problem.
150614  1:37:20 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.51a-24+lenny2-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)

2、查看mysql进程
ps aux |grep mysql
root      2011  0.0  0.0  17332  1468 ?        S    20:34   0:00 /bin/sh /usr/bin/mysqld_safe
mysql     2042 75.9 32.0 11650168 10559232 ?   Rl   20:34  15:15 /usr/sbin/mysqld --basedir=/usr --datadir=/opt/mysql --user=mysql --pid-file=/opt/mysql/1.1.1.1.pid --skip-external-locking --open-files-limit=8192 --port=3306 --socket=/var/run/mysqld/mysqld.sock

3、查看mysql端口(之前有问题的时候端口没在监听中 这个是正常后的)
netstat -tupnl|grep 3306
tcp        0      0 1.1.1.1:3306       0.0.0.0:*               LISTEN      29941/mysqld

4、查看mysql socket(之前有问题的时候socket也没有的 这个是正常后的)
ll /var/run/mysqld/mysqld.sock
srwxrwxrwx 1 mysql mysql 0 2015-06-14 01:37 /var/run/mysqld/mysqld.sock
一共花费了5个小时修复数据 终于正常了。