oracle数据库连接hang，出现了两个listener

ypmqqvbkey

oracle数据库连接hang，出现了两个listener，一个是另一个子进程
处理过程如下
1，出现问题时，用户连接hang住，出现两个listener，现有连接无问题
$ ps -ef|grep lsnr
oracle 26327 1 0 12月 24 ? 5:34 /oracle/product/10.2.0.1/bin/tnslsnr LISTENER -inherit
oracle 16352 12931 1 10:44:26 pts/4 0:00 grep lsnr
oracle 28413 26327 0 21:40:46 ? 0:00 /oracle/product/10.2.0.1/bin/tnslsnr LISTENER -inherit
patrol 29576 29575 0 21:45:19 ? 0:00 /oracle/product/10.2.0.1/bin/lsnrctl stat LISTENER

2，为快速恢复问题，杀掉进程
$ kill -9 26327

3，重启
$ lsnrctl start

LSNRCTL for HPUX: Version 10.2.0.1.0 - Production on 26-DEC-2012 10:44:54

Copyright (c) 1991, 2005, Oracle. All rights reserved.

Starting /oracle/product/10.2.0.1/bin/tnslsnr: please wait...

TNSLSNR for HPUX: Version 10.2.0.1.0 - Production
System parameter file is /oracle/product/10.2.0.1/network/admin/listener.ora
Log messages written to /oracle/product/10.2.0.1/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rps02)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rps02)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for HPUX: Version 10.2.0.1.0 - Production
Start Date 26-DEC-2012 10:44:56
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP ON
Listener Parameter File /oracle/product/10.2.0.1/network/admin/listener.ora
Listener Log File /oracle/product/10.2.0.1/network/log/listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rps02)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

4，恢复正常，只有一个listener
$
$ ps -ef|grep lsnr
oracle 16503 1 0 10:44:56 ? 0:00 /oracle/product/10.2.0.1/bin/tnslsnr LISTENER -inherit
oracle 17637 12931 1 10:48:17 pts/4 0:00 grep lsnr
$

5，后续查询metalink，基本确定为bug，需打oracle Patch 4518443
详见文件Intermittent TNS Listener Hang, New Child Listener Process Forked [ID 340091.1]
除了打补丁，oracle给出的其它解决方案是
a,修改listener.ora文件
SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF
b,移动ons.config文件
cd $ORACLE_HOME/opmn/conf
mv ons.config ons.config.ori
6，更详细的原因见参考metalink文档
In this Document
Description
Occurrence
Symptoms
Workaround
Patches
History
References

Applies to:
Oracle Net Services - Version 10.1.0.3.0 to 10.2.0.2.0 [Release 10.1 to 10.2]
Information in this document applies to any platform.
All new connections via TNS listener hang, no errors reported

Checked for relevance on 05-FEB-2010.
***Checked for relevance on 27-nov-2012***


Description

Intermittently the TNS listener hangs and new connections to the database are not possible.



Occurrence

The issue is that the TNS listener can hang under load if a second spawned listener process is not closed (remains persistent). Secondary listener processes are not unusual, depending on traffic as well as when the OS grep snapshot is taken. However, a persistentsecondary process (longer than say 5 second) is not normal and may be a result of this referenced problem.

TNS listener can hang at any time and effect standalone or RAC systems
Symptoms

Listener process can also consume high amount of CPU

Child TNS listener process is seen when doing a ps on the listener process, eg.:
$ ps -ef | grep tnslsnr

ora10g 8909 1 0 Sep 15 ? 902:44 /u05/10GHOME/DBHOME/bin/tnslsnr sales -inherit
ora10g 22685 8909 0 14:19:23 ? 0:00 /u05/10GHOME/DBHOME/bin/tnslsnr sales -inherit

Killing the child process allows new connections to work until the problem reoccurs




Workaround


Issue is fixed in 10.2.0.3 Patch Set
Oracle Support recommends patching to 10.2.0.5 as this the lastest release

- OR -

Apply Patch 4518443 for the problem (if a patch is available)

- OR -

As a workaround, two steps should be done:

1. The following parameter can be added to listener.ora
SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

Where <listener_name> should be replaced with the actual listener name configured in the LISTENER.ORA file. This parameter is to be placed by itself on an empty line / at the end of file.

For example, if the listener name is LISTENER (default), the parameter would be:
SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF


2. Locate the ons.config file in the 10g(rdbms) home and rename it to something else.
For example:
cd $ORACLE_HOME/opmn/conf
mv ons.config ons.config.orig

The listener needs to be restarted after these changes.

This will both prevent the listener from registering against ONS (Oracle Notification Services), which is the area affected by bug:4518443, as well as disable ONS itself. For more information on ONS, please refer to the specific Oracle documentation, for example,for 10.2, see the Oracle10g Release 2 Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide.

Please note, that adding the SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name> to listener.ora file on RAC and disabling the ONS file, will mean that FAN (fast application notification) will not be possible. See Note 220970.1 RAC: Frequently Asked Questions forfurther information on FAN. Therefore, if you have a RAC configuration, then apply the patch and do not disable ONS or FAN.

Also, please note that this might happen with ANY 10g installation, whether it is RAC related or not, and whether there is an Oracle Application installation or not.



Patches

Apply Patch 4518443 for the problem (if a patch is available)

参考文档：
《Intermittent TNS Listener Hang, New Child Listener Process Forked [ID 340091.1]》