Oracle 10203 RAC环境添加新节点（五）

所向无敌

　　简单描述一下，在Oracle 10203 for Solaris sparc的RAC双节点环境中，新增一个节点的过程。共享存储已经在第三个节点上配置完成，这里主要介绍操作系统上和Oracle上的配置。
　　简单描述添加节点的软件安装过程中碰到的问题。
　　第一个错误是在运行ADDNODE.SH工具时碰到的，在节点3的安装过程中出现了空间不足的错误：
　　INFO: ld.so.1: lsnodes: fatal: libskgxn2.so: open failed: No such file or directory
　　INFO: Vendor clusterware is not detected.
　　INFO: Assigning node number '3' to node 'racnode3'
　　INFO: Setting variable 'CLUSTER_NEW_HOST_NAMES' to 'racnode3,'. Received the value from a code block.
　　INFO: Setting variable 'CLUSTER_NEW_NODE_NUMBERS' to '3,'. Received the value from a code block.
　　INFO: Initializing OUI access setup
　　INFO: Performing operation for OUI access setup
　　INFO:
　　*** Cluster Node Addition Summary Page***
　　m_asNodeArray:racnode2,racnode3,racnode1
　　m_sLocalNode:racnode2
　　sNativeVolName:/data1/
　　m_asNodeArray:racnode2,racnode3,racnode1
　　m_sLocalNode:racnode2
　　SEVERE: oracle.sysman.oii.oiip.oiipg.OiipgRemoteOpsException: Error in checking free space on path '/data1' on node 'racnode3' .
　　at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.throwRemoteOpsException(OiipgClusterOps.java:2456)
　　at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.getFreeSpaceOnNode(OiipgClusterOps.java:2236)
　　at oracle.sysman.oii.oiif.oiifp.OiifpAddNodeSummaryPanel.addAddNodeSpaceInfos(OiifpAddNodeSummaryPanel.java:439)
　　at oracle.sysman.oii.oiif.oiifp.OiifpAddNodeSummaryPanel.populateTree(OiifpAddNodeSummaryPanel.java:246)
　　at oracle.sysman.oii.oiif.oiifp.OiifpAddNodeSummaryPanel.init(OiifpAddNodeSummaryPanel.java:227)
　　at oracle.sysman.oii.oiif.oiifp.OiifpAddNodeSummaryPanel.createPanelElements(OiifpAddNodeSummaryPanel.java:148)
　　at oracle.sysman.oii.oiif.oiifm.OiifmGraphicPageHandler.createWizardPanel(OiifmGraphicPageHandler.java:583)
　　at oracle.sysman.oii.oiif.oiifm.OiifmGraphicPageHandler.showWizardPage(OiifmGraphicPageHandler.java:241)
　　at oracle.sysman.oii.oiif.oiifm.OiifmGraphicPageHandler.processWizardPage(OiifmGraphicPageHandler.java:193)
　　at oracle.sysman.oii.oiif.oiifm.OiifmGraphicSummaryPageHandler.processWizardPage(OiifmGraphicSummaryPageHandler.java:90)
　　at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.doWizardPageOperation(OiifmGraphicInterfaceManager.java:352)
　　at oracle.sysman.oii.oiif.oiifb.OiifbWizChainDlgElem.doOperation(OiifbWizChainDlgElem.java:690)
　　at oracle.sysman.oii.oiif.oiifw.OiifwAddNodeSummaryWCDE.doOperation(OiifwAddNodeSummaryWCDE.java:176)
　　at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
　　at oracle.sysman.oii.oiic.OiicPullSession.doOperation(OiicPullSession.java:1273)
　　at oracle.sysman.oii.oiic.OiicSessionWrapper.doOperation(OiicSessionWrapper.java:289)
　　at oracle.sysman.oii.oiic.OiicInstaller.run(OiicInstaller.java:546)
　　at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:929)
　　at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:866)
　　Caused by: oracle.ops.mgmt.cluster.ClusterException: PRKC-1031 : Error checking free space for /data1 on racnode3
　　df: (/data1    ) ????????????????????????????????
　　at oracle.ops.mgmt.cluster.ClusterCmd.getFreeSpace(ClusterCmd.java:1697)
　　at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.getFreeSpaceOnNode(OiipgClusterOps.java:2218)
　　... 17 more
　　解决了空间问题后，图形安装工具自动解除了错误。
　　第二个问题是在运行root.sh脚本的过程中，安装结束后，需要在新节点上运行root.sh，不过运行出现了下面的错误：
　　# . /data/oracle/product/10.2/crs/root.sh
　　WARNING: directory '/data/oracle/product/10.2' is not owned by root
　　WARNING: directory '/data/oracle/product' is not owned by root
　　WARNING: directory '/data/oracle' is not owned by root
　　WARNING: directory '/data' is not owned by root
　　Checking to see if Oracle CRS stack is already configured
　　Oracle CRS stack is already configured and will be running under init(1M)
　　检查后发现，是由于手工拷贝Oracle目录的时候，将/var/opt/oracle目录也拷贝过来，导致Oracle检查ocr.loc发现已经配置过CRS了，删除ocr.loc后，再次运行root.sh脚本，恢复正常。
　　# . /data/oracle/product/10.2/crs/root.sh
　　WARNING: directory '/data/oracle/product/10.2' is not owned by root
　　WARNING: directory '/data/oracle/product' is not owned by root
　　WARNING: directory '/data/oracle' is not owned by root
　　WARNING: directory '/data' is not owned by root
　　Checking to see if Oracle CRS stack is already configured
　　OCR LOCATIONS =  /dev/rac/ocr
　　Setting the permissions on OCR backup directory
　　Setting up NS directories
　　Oracle Cluster Registry configuration upgraded successfully
　　WARNING: directory '/data/oracle/product/10.2' is not owned by root
　　WARNING: directory '/data/oracle/product' is not owned by root
　　WARNING: directory '/data/oracle' is not owned by root
　　WARNING: directory '/data' is not owned by root
　　clscfg: EXISTING configuration version 3 detected.
　　clscfg: version 3 is10GRelease 2.
　　Successfully accumulated necessary OCR keys.
　　Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
　　node :   
　　node 1: racnode1 racnode1-priv racnode1
　　node 2: racnode2 racnode2-priv racnode2
　　clscfg: Arguments check out successfully.
　　NO KEYS WERE WRITTEN. Supply -force parameter to override.
　　-force is destructive and will destroy any previous cluster
　　configuration.
　　Oracle Cluster Registry for cluster has already been initialized
　　Startup will be queued to init within 30 seconds.
　　Adding daemons to inittab
　　Expecting the CRS daemons to be up within 600 seconds.
　　CSS is active on these nodes.
　　racnode1
　　racnode2
　　racnode3
　　CSS is active on all nodes.
　　Waiting for the Oracle CRSD and EVMD to start
　　Oracle CRS stack installed and running under init(1M)
　　Running vipca(silent) for configuring nodeapps
　　IP地址"racnode1-vip"已被使用。请输入一个未使用的IP地址。
　　不过这里又出现了第三个错误。由于以前存在的实例现在处于启动状态，VIP已经成功分配，所以这里出现了这个问题，下面用oracle用户切换为root权限，启动vipca进行配置：
　　$ su root
　　Password:
　　# ./vipca
　　成功配置后，问题解决。
　　随后碰到的错误仍然和COPY目录有关，由于手工将/data/oracle/oraInventroy目录拷贝到了目标服务器上，因此无论是cluster软件还是database软件安装过程的attach操作都会报错：
　　INFO: Invoking OUI on cluster nodes racnode3
　　INFO: /data1/data/oracle/product/10.2/crs/oui/bin/../bin/runInstaller  -paramFile /data1/data/oracle/product/10.2/crs/oui/bin/../clu
　　sterparam.ini  -silent -ignoreSysPrereqs -attachHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs ORACLE_HOME_NAME=Or
　　aCrs10g_home1 CLUSTER_NODES=racnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode3
　　-remoteInvocation -invokingNodeName racnode2 -logFilePath "/data/oracle/oraInventory/logs" -timestamp 2009-01-30_01-01-56AM
　　SEVERE: oracle.sysman.oii.oiip.oiipg.OiipgRemoteOpsException: Error occured while trying to run Unix command /data1/data/oracle/prod
　　uct/10.2/crs/oui/bin/../bin/runInstaller  -paramFile /data1/data/oracle/product/10.2/crs/oui/bin/../clusterparam.ini  -silent -ignor
　　eSysPrereqs -attachHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs ORACLE_HOME_NAME=OraCrs10g_home1 CLUSTER_NODES=r
　　acnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode3 -remoteInvocation -invokingNo
　　deName racnode2 -logFilePath "/data/oracle/oraInventory/logs" -timestamp 2009-01-30_01-01-56AM on nodes racnode3. [/usr/bin/sh: /dat
　　a1/data/oracle/product/10.2/crs/oui/bin/../bin/runInstaller: ?????? :failed]
　　.
　　.
　　.
　　at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:866)
　　Caused by: oracle.ops.mgmt.cluster.ClusterException: /usr/bin/sh: /data1/data/oracle/product/10.2/crs/oui/bin/../bin/runInstaller: ?
　　????? :failed
　　at oracle.ops.mgmt.cluster.ClusterCmd.runCmd(ClusterCmd.java:1959)
　　at oracle.sysman.oii.oiip.oiipg.OiipgClusterRunCmd.runCmdOnUnix(OiipgClusterRunCmd.java:256)
　　... 13 more
　　SEVERE: Remote 'AttachHome' failed on nodes: 'racnode3'. Refer to '/data/oracle/oraInventory/logs/addNodeActions2009-01-30_01-01-56A
　　M.log' for details.
　　You can manually re-run the following command on the failed nodes after the installation:
　　/data/oracle/product/10.2/crs/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs ORACLE_H
　　OME_NAME=OraCrs10g_home1 CLUSTER_NODES=racnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NOD
　　E=.
　　DATABASE软件的安装过程碰到同样的问题，如果手工执行命令错误如下：
　　$ runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs ORACLE_HOME_NAME=OraCrs10g_home1 CLUSTER_NODES=racnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode3
　　正在启动Oracle Universal Installer...
　　正在检查安装程序要求...
　　检查操作系统版本:必须是5.8, 5.9 or 5.10。   实际为5.10
　　通过
　　检查临时空间:必须大于250 MB。  实际为4149 MB   通过
　　检查交换空间:必须大于500 MB。  实际为4289 MB   通过
　　所有安装程序要求均已满足。
　　OUI-10197:无法在/data/oracle/product/10.2/crs创建新的Oracle主目录。此位置已存在Oracle主目录。请选择其它位置。
　　bash-3.00$ pwd
　　/data/oracle/product/10.2/crs/oui/bin
　　bash-3.00$ cd ../../../database/oui/bin/
　　bash-3.00$ runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/database ORACLE_HOME_NAME=OraDb10g_home1 CLUSTER_NODES=racnode1,racnode2,racnode3 "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode3
　　正在启动Oracle Universal Installer...
　　在oraparam.ini中未找到先决条件检查,不执行系统先决条件检查。
　　The inventory pointer is located at /var/opt/oracle/oraInst.loc
　　The inventory is located at /data/oracle/oraInventory
　　OUI-10197:无法在/data/oracle/product/10.2/database创建新的Oracle主目录。此位置已存在Oracle主目录。请选择其它位置。
　　'AttachHome'失败。
　　这是由于拷贝的oraInvertory目录已经包含了CLUSTER和ORACLE软件信息，因此添加会失败，这里简单的忽略这个错误即可。
　　随后出现了UPDATENODELIST的错误，情况类似：
　　INFO: Invoking OUI on cluster nodes racnode1
　　INFO: /data1/data/oracle/product/10.2/crs/oui/bin/../bin/runInstaller  -paramFile /data1/data/oracle/product/10.2/crs/oui/bin/../clu
　　sterparam.ini  -silent -ignoreSysPrereqs -updateNodeList -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs CLUSTER_NODES=r
　　acnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode1 -remoteInvocation -invokingNo
　　deName racnode2 -logFilePath "/data/oracle/oraInventory/logs" -timestamp 2009-01-30_01-01-56AM
　　SEVERE: oracle.sysman.oii.oiip.oiipg.OiipgRemoteOpsException: Error occured while trying to run Unix command /data1/data/oracle/prod
　　uct/10.2/crs/oui/bin/../bin/runInstaller  -paramFile /data1/data/oracle/product/10.2/crs/oui/bin/../clusterparam.ini  -silent -ignor
　　eSysPrereqs -updateNodeList -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs CLUSTER_NODES=racnode1,racnode2,racnode3 CRS
　　=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode1 -remoteInvocation -invokingNodeName racnode2 -logFilePath
　　"/data/oracle/oraInventory/logs" -timestamp 2009-01-30_01-01-56AM on nodes racnode1. [Starting Oracle Universal Installer...Checking
　　installer requirements...Checking operating system version: must be 5.6, 5.7, 5.8 or 5.9.    Actual 5.8
　　PassedChecking Temp space: must be greater than 250 MB.   Actual 8591 MB    PassedChecking swap space: must be greater tha
　　n 500 MB.   Actual 11377 MB    PassedAll installer requirements met.Checking operating system patches: 108652-66,108921-16,108940-53
　　,108773-18,111310-01,109147-24,111308-03,111111-03,112396-02,110386-03,111023-02,108987-13,108528-21,108989-02,108993-18Checking for
　　patch 108652-66; found 108652-98 obsoleted Checking for patch 108921-16; found Checking for patch 108940-53; found Checking for pat
　　ch 108773-18; found Checking for patch 111310-01; found Checking for patch 109147-24; found Checking for patch 111308-03; found Chec
　　king for patch 111111-03; found Checking for patch 112396-02; found Checking for patch 110386-03; found Checking for patch 111023-02
　　; found Checking for patch 108987-13; found Checking for patch 108528-21; found Checking for patch 108989-02; found Checking for pat
　　ch 108993-18; found     Passed'UpdateNodeList' failed. :failed]
　　.
　　.
　　.
　　at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:866)
　　Caused by: oracle.ops.mgmt.cluster.ClusterException: /usr/bin/sh: /data1/data/oracle/product/10.2/crs/oui/bin/../bin/runInstaller: ?
　　????? :failed
　　at oracle.ops.mgmt.cluster.ClusterCmd.runCmd(ClusterCmd.java:1959)
　　at oracle.sysman.oii.oiip.oiipg.OiipgClusterRunCmd.runCmdOnUnix(OiipgClusterRunCmd.java:256)
　　... 13 more
　　SEVERE: Remote 'UpdateNodeList' failed on nodes: 'racnode1,racnode3'. Refer to '/data/oracle/oraInventory/logs/addNodeActions2009-01
　　-30_01-01-56AM.log' for details.
　　You can manually re-run the following command on the failed nodes after the installation:
　　/data/oracle/product/10.2/crs/oui/bin/runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs CLUS
　　TER_NODES=racnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=.
　　INFO: User Selected: Yes/OK
　　无论是数据库还是CLUSTER的UPDATENODELIST操作也都失败了，不过随后无论是在racnode1上，还是在racnode3上指向的updatenodelist操作都成功了：
　　$ runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs CLUSTER_NODES=racnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode1
　　Starting Oracle Universal Installer...
　　Checking installer requirements...
　　Checking operating system version: must be 5.8, 5.9 or 5.10.    Actual 5.8
　　Passed
　　Checking Temp space: must be greater than 150 MB.   Actual 8639 MB    Passed
　　Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
　　All installer requirements met.
　　'UpdateNodeList' was successful.
　　下面是racnode3上分别执行cluster和database的updatenodelist操作：
　　$ runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs CLUSTER_NODES=racnode1,racnode2,racnode3 CRS=true  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode3
　　正在启动Oracle Universal Installer...
　　正在检查安装程序要求...
　　检查操作系统版本:必须是5.8, 5.9 or 5.10。   实际为5.10
　　通过
　　检查临时空间:必须大于250 MB。  实际为4149 MB   通过
　　检查交换空间:必须大于500 MB。  实际为4289 MB   通过
　　检查监视器:监视器配置至少必须显示256种颜色。   实际为16777216   通过
　　所有安装程序要求均已满足。
　　'UpdateNodeList'成功。
　　$ /runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/database CLUSTER_NODES=racnode1,racnode2,racnode3 CRS=false  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode3
　　bash: /runInstaller:无此文件或目录
　　bash-3.00$ runInstaller -updateNodeList -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/database CLUSTER_NODES=racnode1,racnode2,racnode3 CRS=false  "INVENTORY_LOCATION=/data/oracle/oraInventory" LOCAL_NODE=racnode3
　　正在启动Oracle Universal Installer...
　　在oraparam.ini中未找到先决条件检查,不执行系统先决条件检查。
　　The inventory pointer is located at /var/opt/oracle/oraInst.loc
　　The inventory is located at /data/oracle/oraInventory
　　'UpdateNodeList'成功。
　　通过手工运行，UPDATENODELIST操作成功执行。
　　oracle视频教程请关注:http://u.youku.com/user_video/id_UMzAzMjkxMjE2.html