hp-ux: CRS Installation Failed at VIPCA with CRS-184 Cannot communicate with the

542505989

　　In this Document
  Symptoms
  Changes
  Cause
  Solution
  References

　　
　　
Applies to:
　　 Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.1.0.7 - Release: 10.2 to 11.1
HP-UX PA-RISC (64-bit)
HP-UX Itanium

Symptoms
　　Fresh installation of CRS 10.2.0.1 on a multi node HP-UX RAC cluster.
Running root.sh succeeds on node 1, but fails on the last node while executing vipca.
This is caused by the use of IP address range 10.x.x.x range for the public interface - as documented in note 316583.1.
Whenrunning vipca manually as the root user (as per above document) it fails with error 'CRS-184:  Cannot communicate with the CRS daemon', it did create the VIP resource for the 1st node but neither the GSD nor ONSresources and no nodeapps resources for the other node(s) were created.

The command 'crs_stat -t' works fine, but 'srvctl status nodeapps' or 'srvctl add nodeapps' commands fail with either:


CRS-184 Cannot communicate with CRS Daemon　　 or


PRKO-2105 Error in checking condition of VIP of node: racnode1　　
Basically none of the 'srvctl' commands work while 'crsctl check crs' or 'crs_stat -t' do work.

Changes
　　New installation.
Cause
　　Running 'tusc' against the 'srvctl' command


tusc -faieo /tmp/srvctl.trc srvctl status nodeapss -n racnode1
　　shows the following failure at:


setsockopt(5, SOL_SOCKET, SO_REUSEADDR, 0x9fffffffffffc788, 4) ................. = 0
bind(5, 0x6000000000304480, 16) ................................................ ERR#227 EADDRNOTAVAIL
...
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, 0x9fffffffffffbf68, 4) ................. = 0
bind(3, 0x60000000001f8280, 16) ................................................ ERR#227 EADDRNOTAVAIL
　　
The tusc output shows a number of successful bind()  calls, but eventually bind() starts failing with above error.
All the socket files (permission) in /tmp/.oracle are fine.
The problem in this particular case was caused by an incorrect setting for the udp port range, the parameter udp_smallest_anon_portin /etc/rc.config.d/nddconf was accidentally set to 65500 (from default49152, it is supposed to be set to 9000 as per documentation) while udp_largest_anon_port remained at 65535.
As a result only 35 anonymous ports were available for use, which led to the failure of the srvctl and vipca commands.



Please note that above parameter as well as the configuration file are specific to HP-UX, but do have equivalentson other platforms., so the issue may occur on other platforms as well.　　

Solution
　　1. Engage the system administrator to make sure the file /etc/rc.config.d/nddconf contains the correct values - e.g.:



TRASNSPORT_NAME[3]=udp
NDD_NAME[3]=udp_smallest_anon_port
NDD_VALUE[3]=9000
#
TRASNSPORT_NAME[4]=udp
NDD_NAME[4]=udp_largest_anon_port        
NDD_VALUE[4]=65500　　

2. Verify the actual values for above parameters and correct them dynamically on all nodes to proceed with the installation (if necessary):


# ndd -get /dev/udp udp_smallest_anon_port
# ndd -get /dev/udp udp_largest_anon_port
# ndd -set /dev/udp udp_smallest_anon_port 9000
# ndd -set /dev/udp udp_largest_anon_port 65500
　　
3. Remove any incomplete configuration:


# srvctl remove nodeapps -n <racnode>
　　
4. Rerun vipca as root user and continue with the OUI installer to complete the installation.



Note:
From11.2 onwards, cluvfy will check for above udp port setting, but it is set as ignorable (with -verbose), so it still shows that the parameter check passed.

Check: Kernel parameter for "udp_smallest_anon_port"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode1   65500      9000    failed (ignorable)
racnode2   65500      9000    failed (ignorable)
Result: Kernel parameter check passed for "udp_smallest_anon_port"
Check: Kernel parameter for "udp_largest_anon_port"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
racnode1   65535     65500    failed (ignorable)
racnode2   65535     65500    failed (ignorable)
Result: Kernel parameter check passed for "udp_largest_anon_port"
Prior to that version, please ensure the result is checked manually and above parameters are set correctly.
However:
Due to unpublished, HP-UX specific Bug 9971592these parameters currently are not being corrected by the 11.2 fixup script. This will be fixed in 11.2.0.3 where the fixup script will set the kernel parameters as per installation requirement.