设为首页 收藏本站
查看: 1523|回复: 0

[经验分享] hadoop初探--性能分析1

[复制链接]

尚未签到

发表于 2018-11-1 09:23:34 | 显示全部楼层 |阅读模式
  研究Hadoop也有2个星期了,由于之前有MooseFS的基础,所以学习Hadoop的HDFS部分还算比较顺利。我的环境是非常典型的Hadoop环境,一台NameNode+Hmaster,4台DataNode+RegionServer。如下图所示
DSC0000.png

  说明: DataNode1和DataNode2是VM,配置4VCPU和4G内存,虚拟化平台是ESXI4.1。DataNode3和DataNode4是物理机,OS为Ubuntu11.04,2cpu和2G内存
  实验:利用hadoop自身提供的程序,利用MapReduce计算“TestTable”这个表的“行数”
  [root@master bin]# ./hadoop jar /root/hbase/hbase-0.90.4.jar rowcounter TestTable
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:host.name=master
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_26
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
  。。。。。。。。。。。。。。。。。。。。。。
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/root/hadoop/bin/../lib/native/Linux-i386-32
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:java.compiler=
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-164.el5
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:user.name=root
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Client environment:user.dir=/root/hadoop-0.20.2/bin
  11/10/25 10:34:36 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=datanode2:2181,datanode1:2181,master:2181 sessionTimeout=180000 watcher=hconnection
  11/10/25 10:34:36 INFO zookeeper.ClientCnxn: Opening socket connection to server datanode2/172.20.10.193:2181
  11/10/25 10:34:36 INFO zookeeper.ClientCnxn: Socket connection established to datanode2/172.20.10.193:2181, initiating session
  11/10/25 10:34:36 INFO zookeeper.ClientCnxn: Session establishment complete on server datanode2/172.20.10.193:2181, sessionid = 0x23338df2f890004, negotiated timeout = 180000
  11/10/25 10:34:37 INFO mapred.JobClient: Running job: job_201110241323_0001
  11/10/25 10:34:38 INFO mapred.JobClient: map 0% reduce 0%
  11/10/25 10:35:34 INFO mapred.JobClient: map 2% reduce 0%
  11/10/25 10:39:30 INFO zookeeper.ClientCnxn: Socket connection established to datanode1/172.20.10.201:2181, initiating session
  11/10/25 10:40:05 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x23338df2f890004, likely server has closed socket, closing socket connection and attempting reconnect
  11/10/25 10:40:06 INFO zookeeper.ClientCnxn: Opening socket connection to server datanode2/172.20.10.193:2181
  11/10/25 10:40:06 INFO zookeeper.ClientCnxn: Socket connection established to datanode2/172.20.10.193:2181, initiating session
  11/10/25 10:40:06 INFO zookeeper.ClientCnxn: Session establishment complete on server datanode2/172.20.10.193:2181, sessionid = 0x23338df2f890004, negotiated timeout = 180000
  11/10/25 10:40:13 INFO mapred.JobClient: map 7% reduce 0%
  11/10/25 10:42:31 INFO mapred.JobClient: map 10% reduce 0%
  11/10/25 10:43:04 INFO mapred.JobClient: map 13% reduce 0%
  11/10/25 10:53:08 INFO mapred.JobClient: map 15% reduce 0%
  。。。。。。。。。。。。。。。
  11/10/25 11:11:28 INFO mapred.JobClient: map 36% reduce 0%

  11/10/25 11:18:00 INFO mapred.JobClient: Task>  Task attempt_201110241323_0001_m_000008_0 failed to report status for 608 seconds. Killing!
  Task attempt_201110241323_0001_m_000008_0 failed to report status for 611 seconds. Killing!

  11/10/25 11:18:36 INFO mapred.JobClient: Task>  Task attempt_201110241323_0001_m_000009_0 failed to report status for 646 seconds. Killing!
  Task attempt_201110241323_0001_m_000009_0 failed to report status for 617 seconds. Killing!

  11/10/25 11:18:37 INFO mapred.JobClient: Task>  Task attempt_201110241323_0001_m_000013_0 failed to report status for 624 seconds. Killing!
  Task attempt_201110241323_0001_m_000013_0 failed to report status for 604 seconds. Killing!
  11/10/25 11:26:17 INFO mapred.JobClient: map 39% reduce 0%
  11/10/25 11:26:48 INFO mapred.JobClient: map 42% reduce 0%

  11/10/25 11:35:48 INFO mapred.JobClient: Task>  Task attempt_201110241323_0001_m_000014_0 failed to report status for 600 seconds. Killing!
  11/10/25 11:42:07 INFO mapred.JobClient: map 44% reduce 0%
  11/10/25 11:57:43 INFO mapred.JobClient: map 47% reduce 0%
  11/10/25 12:12:16 INFO mapred.JobClient: map 49% reduce 0%
  。。。。。。。。。。。。。。。。。。。。。
  11/10/25 15:45:26 INFO mapred.JobClient: map 97% reduce 0%
  11/10/25 16:16:57 INFO mapred.JobClient: map 100% reduce 0%
  11/10/25 16:16:59 INFO mapred.JobClient: Job complete: job_201110241323_0001
  11/10/25 16:16:59 INFO mapred.JobClient: Counters: 7
  11/10/25 16:16:59 INFO mapred.JobClient: Job Counters
  11/10/25 16:16:59 INFO mapred.JobClient: Rack-local map tasks=21
  11/10/25 16:16:59 INFO mapred.JobClient: Launched map tasks=58
  11/10/25 16:16:59 INFO mapred.JobClient: Data-local map tasks=37
  11/10/25 16:16:59 INFO mapred.JobClient: org.apache.hadoop.hbase.mapreduce.RowCounter$RowCounterMapper$Counters
  11/10/25 16:16:59 INFO mapred.JobClient: ROWS=6003531
  11/10/25 16:16:59 INFO mapred.JobClient: Map-Reduce Framework
  11/10/25 16:16:59 INFO mapred.JobClient: Map input records=6003531
  11/10/25 16:16:59 INFO mapred.JobClient: Spilled Records=0
  11/10/25 16:16:59 INFO mapred.JobClient: Map output records=0
  web监控如下显示
DSC0001.png

  由于篇幅限制,日志输出中有大量略去,经过5h42min的运算,可以看到TestTable这个表总共有6003531行。日志中出现10个错误,基本就2类,
  第一类是“状态报告超时”
  Task attempt_201110251718_0001_m_000014_0 failed to report status for 602 seconds. Killing!
  Task attempt_201110251718_0001_m_000014_0 failed to report status for 600 seconds. Killing!
  第二类是“租约到期”
  org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '7594235277433071278' does not exist at
  org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1847) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at
  由于篇幅限制,关于性能的分析见下一篇文章《hadoop初探--性能分析2》


运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表,该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有,请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意,请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息,一经发现立即删除。若您因此触犯法律,一切后果自负,我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责,亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏,不得用于商业或者其他用途,否则,一切后果均由您自己承担,我们对此不承担任何法律责任
7、如涉及侵犯版权等问题,请您及时通知我们,我们将立即采取措施予以解决
8、联系人Email:admin@iyunv.com 网址:www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址:https://www.yunweiku.com/thread-629204-1-1.html 上篇帖子: Hadoop 配置 下篇帖子: Hadoop原理
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群,最新一手资源尽在官方微信交流群!快快加入我们吧...

扫描微信二维码查看详情

客服E-mail:kefu@iyunv.com 客服QQ:1061981298


QQ群⑦:运维网交流群⑦ QQ群⑧:运维网交流群⑧ k8s群:运维网kubernetes交流群


提醒:禁止发布任何违反国家法律、法规的言论与图片等内容;本站内容均来自个人观点与网络等信息,非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来,其版权均归原作者及其网站所有,我们尊重他人的合法权益,如有内容侵犯您的合法权益,请及时与我们联系进行核实删除!



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表