oracle 10g RAC 节点重启，但是没有记录有效的日志信息--问题诊断

　　oracle 10g RAC 重启，但是没有记录有效的日志信息
　　from:
Oracle? Database Release Notes
10g Release 2 (10.2) for Linux x86-64
B15666-19
____________________________________________________________________
6.15 Configuring Oracle Clusterware Process Monitor Daemon
　　The 10.2.0.4 patch release for Oracle Clusterware on Linux includes
　　the Oracle Clusterware Process Monitor Daemon (oprocd). It is started
　　automatically by Oracle Clusterware to detect system hangs. When it
　　detects a system hang, it restarts the hung node.
　　Review the following configuration information if you have installed
　　the 10.2.0.4 patch set.
　　Oracle has found wide variations in scheduling latencies observed
　　across operating systems and versions of operating systems. Because
　　of these scheduling latencies, the default values for oprocd can be
　　overly sensitive, particularly under heavy system load, resulting in
　　unnecessary oprocd-initiated restarts (false restarts).
　　Oracle recommends that you address scheduling latencies with your
　　operating system vendor to reduce or eliminate them as much as
　　possible, as they can cause other problems.
　　To overcome these scheduling latencies, Oracle recommends that you
　　set the Oracle Clusterware parameter diagwait to the value 13. This
　　setting increases the time for failed nodes to flush final trace
　　files, which helps to debug the cause of a node failure. You must
　　shut down the cluster to change the diagwait setting. However, if you
　　prefer, you can use the default timing threshold for diagwait. In
　　that case, you do not need to perform the procedure documented here.
　　If you require more aggressive failover times to meet more stringent
　　service level requirements, then you should open a service request
　　with Oracle Support to receive advice about how to tune for lower
　　failover settings.
　　Note:
Changing the diagwait parameter requires a clusterwide shutdown. Oracle recommends that you change the diagwait setting either immediately after the initial installation, or during a scheduled outage.

　　Log in as root, and run the following command on all nodes, where
　　CRS_home is the home directory of the Oracle Clusterware
　　installation:
　　# CRS_home/bin/crsctl stop crs
Enter the following command, where CRS_home is the Oracle Clusterware
　　home:
　　# CRS_home/bin/oprocd stop
Repeat this command on all nodes.
　　From one node of the cluster, change the value of the diagwait
　　parameter to 13 seconds by issuing the following command as root:
　　# CRS_home/bin/crsctl set css diagwait 13 -force
Restart the Oracle Clusterware by running the following command on
　　all nodes:
　　# CRS_home/bin/crsctl start crs
Run the following command to ensure that Oracle Clusterware is
　　functioning properly:
　　# CRS_home/bin/crsctl check crs
　　
来自IBM的解释：
Server running AIX with Oracle RAC reboots itself
　　
Technote (troubleshooting)
　　
Problem(Abstract)
Server running AIX with Oracle RAC reboots itself with no warning
　　Symptom
AIX server shuts down and/or reboots.
　　A REBOOT_ID is logged in /var/adm/ras/errlog indicating "SYSTEM
　　SHUTDOWN BY USER" although no shutdown or reboot command was issued
　　by any user.
　　example error message...
　　LABEL: REBOOT_ID
IDENTIFIER: 2BFA76F6
　　Date/Time: Wed Dec 3 08:19:09 2008
Sequence Number: 1447
Machine Id: 0000ABCD1234
Node Id: nodeA
Class: S
Type: TEMP
Resource Name: SYSPROC
　　Description
SYSTEM SHUTDOWN BY USER
　　Probable Causes
SYSTEM SHUTDOWN
　　Detail Data
USER ID
0
0=SOFT IPL 1=HALT 2=TIME REBOOT
0
TIME TO REBOOT (FOR TIMED REBOOT ONLY)
0
　　Cause
Oracle Real Application Clusters (RAC) is known to reboot the
　　operating system with no warning due to configuration of the oprocd
　　daemon
　　
Environment
AIX with Oracle RAC
　　
Diagnosing the problem
Oracle Real Application Clusters (RAC) typically runs a process called oprocd.
　　
The idea of OPROCD is quite straightforward. It’s goal is to provide
　　I/O fencing. Basically oprocd works by setting a timer, then
　　sleeping. If, when it wakes up again and gets scheduled onto cpu, it
　　sees that a longer time has passed than the acceptable margin, oprocd
　　will decide to reboot the node.
　　You can check for the oprocd process with the ps command...
　　# ps -ef | grep oprocd
root 221672 1 0 08:27:44 - 0:00
/u01/crs/oracle/product/10.2.0/crs_1/bin/oprocd run -t 1000 -m 500 -f
　　These options to oprocd are saying -t 1000 (wake up every 1000 ms)
　　and -m 500 (allow up to 500 ms margin of error on the time that
　　oprocd wakes up before rebooting). In other words, if oprocd wakes up
　　after > 1.5 secs it’s going to force a reboot.
　　
　　Resolving the problem
The timeout and margin times are computed from the elements of
　　diagwait and reboot time and it isn't recommended changing them via
　　the init.cssd file, but rather through the command 'crsctl set css
　　diagwait <secs>'.
　　
There is a formula involved in the calculation of the times. For
　　example, if the reboot time is 3 and you submit a diagwait setting of
　　13 you will get -t 1000 -m 10000.
　　# crsctl set css diagwait 13 -force
　　# ps -ef | grep oprocd
root 221672 1 0 08:27:44 - 0:00
/u01/crs/oracle/product/10.2.0/crs_1/bin/oprocd run -t 1000 -m 10000
　　-f
　　You can see that the margin has changed to 10000 ms, that is 10
　　seconds in place of the default 0.5 seconds. This is a 20 fold
　　increase allows oprocd more time to determine if the node needs to be
　　rebooted.
　　IBM recommends the customer contact Oracle Support before modifying
　　this value.
　　IBM and Oracle came to the agreement that a diagwait value of 13 is a
　　suitable value if the best practices are used...
　　
　　http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101513
　　IBM recommends customers follow best practices, and if possible
　　update to AIX 6.1 or AIX 7.1 with current Technology Levels which
　　include the new non-pagable kernel as the preferred corrective
　　action.
　　The Oracle master document can be found here... http://www.oracle.com/technetwork/database/clusterware/overview/rac-aix-system-stability-131022.pdf
　　
ADDENDUM:
　　The following Oracle document provides additional information on the
　　cssdagent process which is related to oprocd...
　　http://docs.oracle.com/cd/E14072_01/rac.112/e10717/intro.htm
The cssdagent process monitors the cluster and provides I/O fencing.
　　This service formerly was provided by Oracle Process Monitor Daemon
　　(oprocd), also known as OraFenceService on Windows. A cssdagent
　　failure results in Oracle Clusterware restarting the node.
　　root 11010182 1 0 18:43:40 - 0:05
　　/GDICMP/oracle/cloud/product/11.2/bin/cssdagent
　　===
　　Additional Oracle processes which are known to reboot AIX include the
　　following which will appear in ps -ef output...
　　octssd.bin reboot
ohasd.bin reboot
crsd.bin reboot
　　
　　
　　Cross reference information Segment Product Component Platform
　　Version Edition
Operating Systems AIX family AIX 5.2, 5.3, 6.1
　　
　　Document information
AIX family
　　
---------------------------------------------------------------------
　　-----------
　　Software version:
Version Independent
　　
---------------------------------------------------------------------
　　-----------
　　Operating system(s):
AIX
　　
---------------------------------------------------------------------
　　-----------
　　Reference #:
T1011228
　　
---------------------------------------------------------------------
　　-----------
　　Modified date:
2011-10-19
Translate my page
Select LanguageChinese SimplifiedChinese
　　TraditionalFrenchGermanItalianJapaneseKoreanPortugueseRussianSpanish