Oracle RAC Node Reboot Problem (fatal status 13)

静恬2202

　　数据库RAC NODE 常发生自我重启， 查看日志 /var/log/messages
　　Apr 12 13:00:39 ordb1 logger: Oracle clsomon failed with fatal status 13.
　　Apr 12 13:00:40 ordb1 logger: Oracle CRS failure. Rebooting for cluster integrity.
　　
　　
　　查询资料参考文档: http://www.franklinfaces.com/wiki/oracle_documentation/oracle11gR1/B28359_01/relnotes.111/b32001/toc.htm
　　
　　5.11Known Issues on Linux x86-64
　　5.11.3Linux GLIBC Bug Causes Clusterware Instability
With Oracle Clusterware releases 10.2.0.3 CRS Bundle Patch #3, 10.2.0.4, 11.1.0.6 and
11.1.0.7, the CSS Daemon, ocssd, is vulnerable to a Linux GLIBC bug that exists in all
Linux distributions supported by Oracle RAC. This bug most commonly appears in
8
　　the CSS Daemon as a segmentation violation (SEGV), with a CSS Daemon log entry
similar to the following:
ERROR:   clssscExit: CSSD signal 11 in thread GMClientListener
The bug also causes an error that is registered in the Linux /var/log/messages file:
Oracle CSSD failure 139
Oracle clsomon failed with fatal status 13
 If you encounter general instability in Oracle Clusterware, such as random reboots,
then you should consider this as an indication of the Linux GLIBC bug, even in the
absence of these error messages.
 If you require further diagnostic tests before you apply the recommended kernel
patches, then refer to OracleMetaLink node 559365.1 for further information.
The following glibc versions are known to have this problem:
■ Red Hat and Oracle Enterprise Linux 4 glibc-2.3.4-2.39 (fixed in
glibc-2.3.4-2.40, glibc-2.3.4-2.41 included in EL4u7)
■ Red Hat and Oracle Enterprise Linux 5 glibc-2.5-18 (fixed in glibc-2.5-20,
glibc-2.5-24 included in EL5u2)
■ SLES10-SP2  glibc-2.4-31.54 (bug 416838 filed for this problem). Contact
Novell for fix availability.
For a description of the glibc bug, refer to the following URLs:
■  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473812
■ https://bugzilla.redhat.com/show_bug.cgi?id=405781
■ http://rhn.redhat.com/errata/RHBA-2008-0083.html
The distribution version or glibc version must be updated before you install Oracle
Clusterware version 10.2.0.3 CRS Bundle Patch #3, 10.2.0.4 or 11.1.0.6 (or later) to avoid
instability, in the form of random node reboots, caused by the glibc bug. For more
details about this issue, refer to OracleMetaLink note 731599.1.
　　
　　
　　比较一下当前系统：
Oracle Database 11g CRS Release 11.1.0.6.0
Enterprise Linux Enterprise Linux AS release 4 (ORACLE For Linux  October Update 6)
 
基本吻合，其他日志又分析不出原因，先打Patch吧.
 
 
1.  下载Enterprise Linux Enterprise Linux AS release 4 (ORACLE For Linux  October Update 8)
      准备好glibc RPM文件, 
 
2.  先升一个NODE，先停数据库
   ordb3-> su - oracle ordb3-> srvctl stop instance -d [dbname] -i [instnacename] ordb3> srvctl stop nodeapps -n [nodename] ordb3-> su - root [root]#             /u01/app/crs/bin/crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped Oracle Clusterware resources Stopping Cluster Synchronization Services. Shutting down the Cluster Synchronization Services daemon.


 


 
    crs_stat -t 查看NODE状态
      
 
3. 升级glibc patch
  
　　[iyunv@ordb3 ~]# cd /root/glibc-patch [iyunv@ordb3 glibc-patch]# ls compat-glibc-2.3.2-95.30.i386.rpm glibc-common-2.3.4-2.43.x86_64.rpm glibc-profile-2.3.4-2.43.x86_64.rpm compat-glibc-2.3.2-95.30.x86_64.rpm glibc-devel-2.3.4-2.43.i386.rpm glibc-utils-2.3.4-2.43.x86_64.rpm compat-glibc-headers-2.3.2-95.30.x86_64.rpm glibc-devel-2.3.4-2.43.x86_64.rpm Up.txt glibc-2.3.4-2.43.i686.rpm glibc-headers-2.3.4-2.43.x86_64.rpm glibc-2.3.4-2.43.x86_64.rpm glibc-kernheaders-2.4-9.1.103.EL.x86_64.rpm [iyunv@ordb3 glibc-patch]# rpm -qa |grep glibc glibc-2.3.4-2.39 compat-glibc-2.3.2-95.30 glibc-headers-2.3.4-2.39 glibc-utils-2.3.4-2.39 glibc-2.3.4-2.39 glibc-devel-2.3.4-2.39 compat-glibc-headers-2.3.2-95.30 glibc-kernheaders-2.4-9.1.100.EL glibc-profile-2.3.4-2.39 glibc-devel-2.3.4-2.39 glibc-common-2.3.4-2.39 compat-glibc-2.3.2-95.30 [iyunv@ordb3 glibc-patch]# rpm -Uvh --force glibc*.rpm warning: glibc-2.3.4-2.43.i686.rpm: V3 DSA signature: NOKEY, key ID b38a8516 error: Failed dependencies: glibc-devel = 2.3.4-2.39 is needed by (installed) nptl-devel-2.3.4-2.39.x86_64 [iyunv@ordb3 glibc-patch]# rpm -e nptl-devel-2.3.4-2.39 [iyunv@ordb3 glibc-patch]# rpm -Uvh --force glibc*.rpm warning: glibc-2.3.4-2.43.i686.rpm: V3 DSA signature: NOKEY, key ID b38a8516 Preparing... ########################################### [100%] 1:glibc-common ########################################### [ 11%] 2:glibc ########################################### [ 22%] Stopping sshd:[ OK ] Starting sshd:[ OK ] 3:glibc warning: /etc/ld.so.conf created as /etc/ld.so.conf.rpmnew warning: /etc/nsswitch.conf created as /etc/nsswitch.conf.rpmnew ########################################### [ 33%] 4:glibc-kernheaders ########################################### [ 44%] 5:glibc-utils ########################################### [ 56%] 6:glibc-headers ########################################### [ 67%] 7:glibc-devel ########################################### [ 78%] 8:glibc-devel ########################################### [ 89%] 9:glibc-profile ########################################### [100%] [iyunv@ordb3 glibc-patch]# rpm -qa |grep glibc compat-glibc-2.3.2-95.30 glibc-headers-2.3.4-2.43 glibc-2.3.4-2.43 glibc-2.3.4-2.43 compat-glibc-headers-2.3.2-95.30 glibc-devel-2.3.4-2.43 glibc-kernheaders-2.4-9.1.103.EL glibc-common-2.3.4-2.43 glibc-profile-2.3.4-2.43 glibc-devel-2.3.4-2.43 compat-glibc-2.3.2-95.30


 
　　4. 用ORACLE 用户 relink
　　
　　relink all >> relink.log 2>> relink.log


　　检查log没有fatal 错误
　　
　　5. reboot 系统 
　　
　　6. 其他NODE 按此步骤2-5 。
　　
 
 
? 系统Patch是否稳定， 需要时间观察
 
30days 后系统稳定