设为首页 收藏本站
查看: 803|回复: 0

[经验分享] hadoop节点挂死的一次分析报表。

[复制链接]

尚未签到

发表于 2017-12-17 22:42:21 | 显示全部楼层 |阅读模式
  hadoop的一个节点unused了。然后重启启动hadoop的服务,仍有有一个hadoop的节点起不来。多次重启hadoop和杀进程之后,发现hadoop的master和slave节点上的状态在切换,没有达到同步起停;当master起来的时候,slave节点上的hadoop就unused,  当slave节点上的hadoop状态为running的时候,master节点上的hadoop节点的状态就是unused.主从没法同步起停。
  在数据库入库的时候,日志里报如下错误:
  这个时候突然想到了hadoop的主从站点之后的关系是不是没有同步好,文件进入了安全模式。
  进入到hadoop的bin目录下,执行退出安全模式的命令,第一次是在hadoop服务停了 之后,执行退出安全模式的命令显示没有连接成功。
  然后启动hadoop,这个时候hadoop下面的都运行正常了,执行退出安全模式。
  退出安全模式命令: hadoop dfsadmin -safemode leave
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.library.path=/opt/oracle/app/client/oracle/product/11.2.0/inoc/lib:/lib:/usr/lib:/usr/java/packages/lib/a

  md64:/usr/lib64:/lib64:/lib:/usr/lib]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.io.tmpdir=/tmp]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:java.compiler=<NA>]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.name=Linux]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.arch=amd64]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:os.version=2.6.32.59-0.9-default]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.name=acrosspm]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.home=/home/acrosspm]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Client environment:user.dir=/opt/netwatcher/pm4h2/work/conf/pmpadmin]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Initiating client connection, connectString=10.215.133.36:15248 sessionTimeout=180000 watcher=hconnection]


[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [The>
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Opening socket connection to server pmweb.site/10.215.133.36:15248. Will not attempt to authenticate

  using SASL (unknown error)]
[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Socket connection established to pmweb.site/10.215.133.36:15248, initiating session]

[2017-06-25 18:03:31] [20170625#171000#500#SDP#1495561082439[3/24]-SendThread(pmweb.site:15248)] [INFO] [PM_DPL_901_00000] [Session establishment complete on server pmweb.site/10.215.133.36:15248, sessionid = 0x35cdf7a17c400

  28, negotiated timeout = 180000]
[2017-06-25 18:04:31] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:05:31] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:06:31] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:07:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:08:31] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:09:31] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:10:34] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:10:54] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:11:56] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:12:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:13:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:14:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:16:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:17:02] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:18:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:19:05] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:20:05] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:20:25] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:21:30] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:22:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:23:33] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:24:36] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:24:56] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:26:01] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:27:10] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:28:13] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:28:33] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:29:38] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:30:41] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:31:41] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:32:44] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:33:47] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:34:50] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:35:50] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:36:50] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:37:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:38:59] [20170625#171000#500#SDP#1495561082439[3/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:39:59] [20170625#170500#500#SDPServiceClass#1495561082439[2/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:40:59] [20170625#171500#500#SDP#1495561082439[5/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:41:59] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:42:59] [20170625#171000#500#SDPServiceClass#1495561082439[4/24]] [INFO] [PM_MOE_100_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:44:02] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [Problem connecting to server: pmapp.site/10.215.133.31:15241]

[2017-06-25 18:44:04] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 0 of 10 failed; retrying after sleep of 1000]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:05] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 1 of 10 failed; retrying after sleep of 1006]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:06] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 2 of 10 failed; retrying after sleep of 1005]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:07] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 3 of 10 failed; retrying after sleep of 2014]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:09] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 4 of 10 failed; retrying after sleep of 2015]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:11] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 5 of 10 failed; retrying after sleep of 4034]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:15] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 6 of 10 failed; retrying after sleep of 4007]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:19] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 7 of 10 failed; retrying after sleep of 8075]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:27] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 8 of 10 failed; retrying after sleep of 16070]

  org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1415)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
  at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
  at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:712)
  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isMasterRunning(HConnectionManager.java:759)
  at com.inspur.pm.backend.core.hbase.HbaseFactory.creatTable(HbaseFactory.java:152)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.getFSConnection(FileEngineImplDistributed.java:412)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.insert2HbaseAndOracle(FileEngineImplDistributed.java:744)
  at com.inspur.pm.backend.core.fileengine.impl.FileEngineImplDistributed.putFile2FSByFileDesc(FileEngineImplDistributed.java:275)
  at com.inspur.pmv5.dpl.pif2pia.service.CommonService.storeFileToHBASE(CommonService.java:97)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.storePIA2HBASE(BussinessLogicDealMain.java:1115)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.generatePIAFile(BussinessLogicDealMain.java:610)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessLogicDealMain.deal(BussinessLogicDealMain.java:414)
  at com.inspur.pmv5.dpl.pif2pia.service.BussinessThread.run(BussinessThread.java:64)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
[2017-06-25 18:44:43] [20170625#170500#500#SDP#1495561082439[1/24]] [INFO] [PM_DPL_901_00000] [getMaster attempt 9 of 10 failed; no more retrying.]

运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表,该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有,请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意,请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息,一经发现立即删除。若您因此触犯法律,一切后果自负,我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责,亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏,不得用于商业或者其他用途,否则,一切后果均由您自己承担,我们对此不承担任何法律责任
7、如涉及侵犯版权等问题,请您及时通知我们,我们将立即采取措施予以解决
8、联系人Email:admin@iyunv.com 网址:www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址:https://www.yunweiku.com/thread-425189-1-1.html 上篇帖子: Hadoop 2.7.3 HA 搭建及遇到的一些问题 下篇帖子: Hadoop通过HCatalog编写Mapreduce任务访问hive库中schema数据
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群,最新一手资源尽在官方微信交流群!快快加入我们吧...

扫描微信二维码查看详情

客服E-mail:kefu@iyunv.com 客服QQ:1061981298


QQ群⑦:运维网交流群⑦ QQ群⑧:运维网交流群⑧ k8s群:运维网kubernetes交流群


提醒:禁止发布任何违反国家法律、法规的言论与图片等内容;本站内容均来自个人观点与网络等信息,非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来,其版权均归原作者及其网站所有,我们尊重他人的合法权益,如有内容侵犯您的合法权益,请及时与我们联系进行核实删除!



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表