EM 12c: OMS Failed to start After restarting the Repository Database or reboot o

32e2

EM 12c: OMS Failed to start After restarting the Repository Database or reboot of the server
今天启动Oms发现了这个问题，启动不了：


[oracle@ora11g64 bin]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server is Down


查看日志：


日志中记录1：
/u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log


2014-09-19 13:57:06,329 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>FAILED_NOT_RESTARTABLE
2014-09-19 13:57:06,330 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>
2014-09-19 13:57:06,331 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>sleep for 10 seconds...
2014-09-19 13:57:16,334 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>status of EMGC_OMS1:
2014-09-19 13:57:16,336 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>
2014-09-19 13:57:16,337 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>FAILED_NOT_RESTARTABLE
2014-09-19 13:57:16,401 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>
2014-09-19 13:57:16,401 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>timed out waiting for EMGC_OMS1 to be RUNNING
2014-09-19 13:57:16,402 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>status of node manager:
2014-09-19 13:57:16,402 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>Currently connected to Node Manager to monitor the domain GCDomain.
2014-09-19 13:57:16,403 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>1
2014-09-19 13:57:16,413 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>Successfully disconnected from Node Manager.
2014-09-19 13:57:16,416 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>_END_
2014-09-19 13:57:16,416 [Thread-2] INFO  wls.OMSController run.1744 - <ERR>_END_
2014-09-19 13:57:16,866 [main] INFO  wls.OMSController processCmd.1392 - wlst process is finished
2014-09-19 13:57:16,867 [main] INFO  wls.OMSController processCmd.1394 - wlst process exited with code 1
2014-09-19 13:57:16,867 [main] INFO  wls.OMSController processCmd.1396 - wlst outstream joined
2014-09-19 13:57:16,868 [main] INFO  wls.OMSController processCmd.1398 - wlst errstream joined
2014-09-19 13:57:16,868 [main] INFO  wls.OMSController processCmd.1406 - process exited with code 1
2014-09-19 13:57:16,880 [main] DEBUG oms.StatusOMSCmd processStatusOMS.159 - HTTP port in emgc property file: 7203
2014-09-19 13:57:16,881 [main] DEBUG oms.StatusOMSCmd processStatusOMS.160 - HTTPS port in emgc property file: 7302
2014-09-19 13:57:16,881 [main] DEBUG oms.StatusOMSCmd processStatusOMS.166 - Webtier HTTP port in emgc property file: 7789
2014-09-19 13:57:16,882 [main] DEBUG oms.StatusOMSCmd processStatusOMS.167 - Webtier HTTPS port in emgc property file: 7801
2014-09-19 13:57:16,882 [main] DEBUG oms.StatusOMSCmd processStatusOMS.186 - Using https port.
2014-09-19 13:57:16,882 [main] DEBUG oms.StatusOMSCmd processStatusOMS.203 - Using webtier https port.
2014-09-19 13:57:17,074 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7302/em/faces/logon/core-uifwk-console-login
2014-09-19 13:57:18,263 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7801/empbs/check
2014-09-19 13:57:18,281 [main] DEBUG oms.StatusOMSCmd processStatusOMS.240 - console page status code is 404
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.242 - pbs page status code is 200
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.245 - console header is null
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.247 - pbs header is null
2014-09-19 13:57:18,330 [main] DEBUG oms.StatusOMSCmd processStatusOMS.287 - Encountered error while hitting page. Status code is 404
2014-09-19 13:57:18,330 [main] INFO  oms.StatusOMSCmd processStatusOMS.294 - Failed to connect to em login page
2014-09-19 13:57:18,330 [main] INFO  oms.StatusOMSCmd processStatusOMS.295 - Encountered error while hitting page. Status codes are 404 for console and 200 for PBS
2014-09-19 13:57:18,331 [main] INFO  oms.StatusOMSCmd processStatusOMS.296 - Connection exception when trying to hit OMS page. OMS is down
2014-09-19 13:57:18,331 [main] INFO  wls.OMSController statusOMS.1354 - statusOMS finished with result: 8
2014-09-19 13:57:18,331 [main] INFO  wls.OMSController startOMS.940 - startOMS finished with result: 1
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.159 - HTTP port in emgc property file: 7203
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.160 - HTTPS port in emgc property file: 7302
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.166 - Webtier HTTP port in emgc property file: 7789
2014-09-19 13:57:18,332 [main] DEBUG oms.StatusOMSCmd processStatusOMS.167 - Webtier HTTPS port in emgc property file: 7801
2014-09-19 13:57:18,333 [main] DEBUG oms.StatusOMSCmd processStatusOMS.186 - Using https port.
2014-09-19 13:57:18,333 [main] DEBUG oms.StatusOMSCmd processStatusOMS.203 - Using webtier https port.
2014-09-19 13:57:18,333 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7302/em/faces/logon/core-uifwk-console-login
2014-09-19 13:57:18,458 [main] DEBUG oms.StatusOMSCmd pingOMSCheck.341 - Attempting to connect to https://ora11g64:7801/empbs/check
2014-09-19 13:57:18,480 [main] DEBUG oms.StatusOMSCmd processStatusOMS.240 - console page status code is 404
2014-09-19 13:57:18,481 [main] DEBUG oms.StatusOMSCmd processStatusOMS.242 - pbs page status code is 200
2014-09-19 13:57:18,481 [main] DEBUG oms.StatusOMSCmd processStatusOMS.245 - console header is null
2014-09-19 13:57:18,481 [main] DEBUG oms.StatusOMSCmd processStatusOMS.247 - pbs header is null
2014-09-19 13:57:18,482 [main] DEBUG oms.StatusOMSCmd processStatusOMS.287 - Encountered error while hitting page. Status code is 404
2014-09-19 13:57:18,482 [main] INFO  oms.StatusOMSCmd processStatusOMS.294 - Failed to connect to em login page
2014-09-19 13:57:18,482 [main] INFO  oms.StatusOMSCmd processStatusOMS.295 - Encountered error while hitting page. Status codes are 404 for console and 200 for PBS
2014-09-19 13:57:18,484 [main] INFO  oms.StatusOMSCmd processStatusOMS.296 - Connection exception when trying to hit OMS page. OMS is down
2014-09-19 13:57:18,499 [main] INFO  wls.OMSController statusOMS.1354 - statusOMS finished with result: 8


日志文件中记:2：
2014-09-16 13:18:38,209 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>Successfully started server EMGC_OMS1 ...
2014-09-16 13:18:38,229 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>new status of EMGC_OMS1:
2014-09-16 13:18:38,229 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>
2014-09-16 13:18:38,229 [Thread-1] INFO  wls.OMSController run.1744 - <OUT>RUNNING


日志中记录3：
[oracle@ora11g64 logs]$ pwd
/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs


Repos details fetched from credstore
Fetched repository credentials from Credential Store
Invalid Connection Pool. ERROR = User credentials doesn‘t match the existing ones
Failed to verify repository
java.lang.NullPointerException




metalink:文档 ID 1459936.1


CAUSE
OMS will not be able to make the connection to the database and restarting also fails, if the background processes are not stopped completely.


SOLUTION
As a workaround, follow the steps.


1. Stop the OMS using


/bin>./emctl stop oms -all


2. Check the following processes


$ps -ef | grep java
$ps -ef | grep opmn
$ps -ef | grep http


Please kill all the processes related to OMS user.


3. Restart the Repository Database.


4. Now start the OMS


OMS_HOME/bin>./emctl start oms



To prevent the issue in future:


Apply the Patch 13466904 to the 12.1.0.1.1 (With BP1) OMS home. After applying the patch, we dont need to restart the OMS if the reposiotry database is restarted.


This issue is fixed in 12.1.0.2




我的环境


[oracle@ora11g64 bin]$ ./emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
[oracle@ora11g64 bin]$ ps -ef |grep java
oracle    4753 4708  2 13:01 ?        00:02:40 /u01/app/oracle/Middleware/jdk16/jdk/bin/java -server -Xms256m -Xmx1740m -XX:MaxPermSize=768M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 -XX:PermSize=128m -Dweblogic.Name=EMGC_OMS1 -Djava.security.policy=/u01/app/oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -DINSTANCE_HOME=/u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1 -DORACLE_HOME=/u01/app/oracle/Middleware/oms -Ddomain.home=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain -Djava.awt.headless=true -Ddomain.name=GCDomain -Doracle.sysman.util.logging.mode=dual_mode -Djbo.doconnectionpooling=true -Djbo.txn.disconnect_level=1 -Docm.repeater.home=/u01/app/oracle/Middleware/oms -Djbo.ampool.minavailablesize=1 -Djbo.ampool.timetolive=-1 -Djbo.load.components.lazily=true -Djbo.recyclethreshold=50 -Djbo.ampool.maxavailablesize=50 -Djava.security.egd=file:///dev/./urandom -Dweblogic.debug.DebugSecurityAtn=true -Dweblogic.debug.DebugWebAppSecurity=true -Dweblogic.SSL.LoginTimeoutMillis=300000 -Djps.auth.debug=true -Djps.authz=ACC -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.policystore.hybrid.mode= -Djps.subject.cache.key=5 -Djps.subject.cache.ttl=600000 -Xbootclasspath/p:/u01/app/oracle/Middleware/oms/sysman/jlib/diagpatch_bug11725986.jar -Djdkpatchlog=/u01/app/oracle/Middleware/oms/sysman/log/diagpatch_bug11725986.log -Doracle.apm.home=/u01/app/oracle/Middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -da -Dplatform.home=/u01/app/oracle/Middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/Middleware/wlserver_10.3/server -Dcommon.components.home=/u01/app/oracle/Middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain -Djrockit.optfile=/u01/app/oracle/Middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/servers/EMGC_OMS1 -Doracle.domain.config.dir=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/app/oracle/Middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/Middleware/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Doracle.apm.home=/u01/app/oracle/Middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dweblogic.management.discover=false -Dweblogic.management.server=https://ora11g64:7102 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/Middleware/patch_wls1035/profiles/default/sysext_manifest_classpath weblogic.Server
oracle   10120  8047  0 14:53 pts/2    00:00:00 grep java
[oracle@ora11g64 bin]$ kill -94753
[oracle@ora11g64 bin]$ ps -ef |grep java
oracle   10147  8047  0 14:54 pts/2    00:00:00 grep java
[oracle@ora11g64 bin]$ ps -ef |grep opmn
oracle   10151  8047  0 14:54 pts/2    00:00:00 grep opmn
[oracle@ora11g64 bin]$ ps -ef |grep http
oracle   10155  8047  0 14:54 pts/2    00:00:00 grep http
[oracle@ora11g64 bin]$ sqlplus / as sysdba


SQL*Plus: Release 11.2.0.1.0 Production on Fri Sep 19 14:54:56 2014


Copyright (c) 1982, 2009, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


SYS@ORA11G>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SYS@ORA11G>startup
ORACLE instance started.


Total System Global Area 2137886720 bytes
Fixed Size                  2215064 bytes
Variable Size             822084456 bytes
Database Buffers         1308622848 bytes
Redo Buffers                4964352 bytes
Database mounted.
Database opened.
SYS@ORA11G>exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@ora11g64 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up




还有一个问题会导致：
就是归档路径下的文件多了  空闲空间不足导致的，删除了后重启就行。
但是我的实验在非归档模式 ，所以不行。


结果果然可以，非常感谢 metalink！