hadoop学习入门之伪分布式部署及测试

maxc1017

安装运行伪分布式Hadoop（以0.20.2版本为例）<wbr style="line-height:25px; font-family:Arial,Helvetica,simsun,u5b8bu4f53; font-size:14px"></wbr>
下载Hadoop：地址：http://www.apache.org/dyn/closer.cgi/hadoop/common/选择一个镜像地址。选择版本。


操作都在hadoop的home目录下。





准备工作



解压所下载的Hadoop发行版。编辑conf/hadoop-env.sh文件，至少需要将JAVA_HOME设置为Java安装根路径。

如下命令：
$ bin/hadoop

将会显示hadoop脚本的使用文档。

用以下三种支持的模式中的一种启动Hadoop集群：

• 单机模式
  
• 伪分布式模式
  
• 完全分布式模式
  

下面介绍伪分布式的配置。




伪分布式模式的操作方法



Hadoop可以在单节点上以所谓的伪分布式模式运行，此时每一个Hadoop守护进程都作为一个独立的Java进程运行。



配置



使用如下的conf/core-site.xml:



<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://192.168.0.101:9000</value>
</property>
</configuration>

conf/hdfs-site.xml:





<configuration>
<property>
  <name>fs.replication</name>
  <value>1</value>
</property>
</configuration>




conf/mapred-site.xml:



<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>192.168.0.101:9001</value>
</property>
</configuration>

[size=0.76em]
首先，请求 namenode 对 DFS 文件系统进行格式化。在安装过程中完成了这个步骤，但是了解是否需要生成干净的文件系统是有用的。


bin/hadoop namenode -format

输出：




11/11/30 09:53:56 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubuntu1/192.168.0.101
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
11/11/30 09:53:56 INFO namenode.FSNamesystem: fsOwner=root,root
11/11/30 09:53:56 INFO namenode.FSNamesystem: supergroup=supergroup
11/11/30 09:53:56 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/11/30 09:53:56 INFO common.Storage: Image file of size 94 saved in 0 seconds.
11/11/30 09:53:57 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
11/11/30 09:53:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu1/192.168.0.101
************************************************************/

执行：bin/start-all.sh


输出：

starting namenode, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-namenode-ubuntu1.out
localhost: starting datanode, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-datanode-ubuntu1.out
localhost: starting secondarynamenode, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-secondarynamenode-ubuntu1.out
starting jobtracker, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-jobtracker-ubuntu1.out
localhost: starting tasktracker, logging to /usr/hadoop-0.20.2/bin/../logs/hadoop-root-tasktracker-ubuntu1.out





检查hdfs ：bin/hadoopfs
-ls /
输出目录文件则正常。
hadoop文件系统操作：
bin/hadoop
fs -mkdir test
bin/hadoop
fs -ls test
bin/hadoop
fs -rmr test
测试hadoop：
bin/hadoop
fs -mkdir input
自己建立两个文本文件：file1和file2放在/opt/hadoop/sourcedata下
执行：bin/hadoopfs
-put/opt/hadoop/sourcedata/file*
input
执行：bin/hadoop
jar hadoop-0.20.2-examples.jar wordcount input output
输出：


11/11/30 10:15:38 INFO input.FileInputFormat: Total input paths to process : 2
11/11/30 10:15:52 INFO mapred.JobClient: Running job: job_201111301005_0001
11/11/30 10:15:53 INFO mapred.JobClient:  map 0% reduce 0%
11/11/30 10:19:07 INFO mapred.JobClient:  map 50% reduce 0%
11/11/30 10:19:14 INFO mapred.JobClient:  map 100% reduce 0%
11/11/30 10:19:46 INFO mapred.JobClient:  map 100% reduce 100%
11/11/30 10:19:54 INFO mapred.JobClient: Job complete: job_201111301005_0001
11/11/30 10:19:59 INFO mapred.JobClient: Counters: 17
11/11/30 10:19:59 INFO mapred.JobClient:   Job Counters
11/11/30 10:19:59 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/30 10:19:59 INFO mapred.JobClient:     Launched map tasks=2
11/11/30 10:19:59 INFO mapred.JobClient:     Data-local map tasks=2
11/11/30 10:19:59 INFO mapred.JobClient:   FileSystemCounters
11/11/30 10:19:59 INFO mapred.JobClient:     FILE_BYTES_READ=146
11/11/30 10:19:59 INFO mapred.JobClient:     HDFS_BYTES_READ=64
11/11/30 10:19:59 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=362
11/11/30 10:19:59 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=60
11/11/30 10:19:59 INFO mapred.JobClient:   Map-Reduce Framework
11/11/30 10:19:59 INFO mapred.JobClient:     Reduce input groups=9
11/11/30 10:19:59 INFO mapred.JobClient:     Combine output records=13
11/11/30 10:19:59 INFO mapred.JobClient:     Map input records=2
11/11/30 10:19:59 INFO mapred.JobClient:     Reduce shuffle bytes=102
11/11/30 10:19:59 INFO mapred.JobClient:     Reduce output records=9
11/11/30 10:19:59 INFO mapred.JobClient:     Spilled Records=26
11/11/30 10:19:59 INFO mapred.JobClient:     Map output bytes=120
11/11/30 10:19:59 INFO mapred.JobClient:     Combine input records=14
11/11/30 10:19:59 INFO mapred.JobClient:     Map output records=14
11/11/30 10:19:59 INFO mapred.JobClient:     Reduce input records=13

执行成功！
其他查看结果命令：



bin/hadoop fs -ls /user/root/outputbin/hadoop fs -cat output/part-r-00000bin/hadoop fs -cat output/part-r-00000 | head -13bin/hadoop fs -get output/part-r-00000 output.txt
cat output.txt | head -5bin/hadoop fs -rmr output

也可以使用浏览器查看，地址：
http://192.168.0.101:50030(mapreduce的web页面)
http://192.168.0.101:50070(hdfs的web页面)
下面执行grep的mapreduce任务：
执行：bin/hadoop
fs -rmr output
执行：bin/hadoop
jar hadoop-0.20.2-examples.jar
grep input output 'hadoop'
输出：


11/11/30 10:28:37 INFO mapred.FileInputFormat: Total input paths to process : 2
11/11/30 10:28:40 INFO mapred.JobClient: Running job: job_201111301005_0002
11/11/30 10:28:41 INFO mapred.JobClient:  map 0% reduce 0%
11/11/30 10:34:16 INFO mapred.JobClient:  map 66% reduce 0%
11/11/30 10:37:40 INFO mapred.JobClient:  map 100% reduce 11%
11/11/30 10:37:50 INFO mapred.JobClient:  map 100% reduce 22%
11/11/30 10:37:54 INFO mapred.JobClient:  map 100% reduce 66%
11/11/30 10:38:15 INFO mapred.JobClient:  map 100% reduce 100%
11/11/30 10:38:30 INFO mapred.JobClient: Job complete: job_201111301005_0002
11/11/30 10:38:32 INFO mapred.JobClient: Counters: 18
11/11/30 10:38:32 INFO mapred.JobClient:   Job Counters
11/11/30 10:38:32 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/30 10:38:32 INFO mapred.JobClient:     Launched map tasks=3
11/11/30 10:38:32 INFO mapred.JobClient:     Data-local map tasks=3
11/11/30 10:38:32 INFO mapred.JobClient:   FileSystemCounters
11/11/30 10:38:32 INFO mapred.JobClient:     FILE_BYTES_READ=40
11/11/30 10:38:32 INFO mapred.JobClient:     HDFS_BYTES_READ=77
11/11/30 10:38:32 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=188
11/11/30 10:38:32 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=109
11/11/30 10:38:32 INFO mapred.JobClient:   Map-Reduce Framework
11/11/30 10:38:32 INFO mapred.JobClient:     Reduce input groups=1
11/11/30 10:38:32 INFO mapred.JobClient:     Combine output records=2
11/11/30 10:38:32 INFO mapred.JobClient:     Map input records=2
11/11/30 10:38:32 INFO mapred.JobClient:     Reduce shuffle bytes=46
11/11/30 10:38:32 INFO mapred.JobClient:     Reduce output records=1
11/11/30 10:38:32 INFO mapred.JobClient:     Spilled Records=4
11/11/30 10:38:32 INFO mapred.JobClient:     Map output bytes=30
11/11/30 10:38:32 INFO mapred.JobClient:     Map input bytes=64
11/11/30 10:38:32 INFO mapred.JobClient:     Combine input records=2
11/11/30 10:38:32 INFO mapred.JobClient:     Map output records=2
11/11/30 10:38:32 INFO mapred.JobClient:     Reduce input records=2
11/11/30 10:38:36 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

执行：bin/hadoop fs -cat output/part-00000
输出：2hadoop
成功完成伪分布式的部署及测试。如有问题，请留言！