linux下安装hadoop步骤

starxzj

　　下面的安装手册是我在hadoop第一版时做的，和现在的hadoop不太符合
一、前期准备:下载hadoop: http://hadoop.apache.org/core/releases.htmlhttp://hadoop.apache.org/common/releases.htmlhttp://www.apache.org/dyn/closer.cgi/hadoop/core/http://labs.xiaonei.com/apache-mirror/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gzhttp://labs.xiaonei.com/apache-mirror/hadoop/二、硬件环境共有3台机器，均使用的CentOS，Java使用的是jdk1.6.0。三、安装JAVA6sudo apt-get install sun-java6-jdk/etc/environment打开之后加入：#中间是以英文的冒号隔开，记得windows中是以英文的分号做为分隔的CLASSPATH=.:/usr/local/java/libJAVA_HOME=/usr/local/java三、配置host表[root@hadoop ~]# vi /etc/hosts127.0.0.1 localhost192.168.13.100 namenode192.168.13.108 datanode1192.168.13.110 datanode2[root@test ~]# vi /etc/hosts127.0.0.1 localhost192.168.13.100 namenode192.168.13.108 datanode1[root@test2 ~]# vi /etc/host127.0.0.1 localhost192.168.13.100 namenode192.168.13.110 datanode2添加用户和用户组addgroup hadoopadduser hadoopusermod -a -G hadoop hadooppasswd hadoop配置ssh:服务端:su hadoopssh-keygen -t rsacp>客户端chmod 700 /home/hadoopchmod 755 /home/hadoop/.sshsu hadoopcd /homemkdir .ssh服务端:chmod 644 /home/hadoop/.ssh/authorized_keysscp authorized_keys datanode1:/home/hadoop/.ssh/scp authorized_keys datanode2:/home/hadoop/.ssh/ssh datanode1ssh datanode2　如果ssh配置好了就会出现以下提示信息The authenticity of host [dbrg-2] can't be established.Key fingerpr is 1024 5f:a0:0b:65:d3:82:df:ab:44:62:6d:98:9c:fe:e9:52.Are you sure you want to continue connecting (yes/no)?　　OpenSSH告诉你它不知道这台主机但是你不用担心这个问题你是第次登录这台主机键入“yes”这将把这台主机“识别标记”加到“~/.ssh/know_hosts”文件中第 2次访问这台主机时候就不会再显示这条提示信不过别忘了测试本机ssh　dbrg-1　mkdir /home/hadoop/HadoopInstalltar -zxvf hadoop-0.20.1.tar.gz -C /home/hadoop/HadoopInstall/cd /home/hadoop/HadoopInstall/ln -s hadoop-0.20.1 hadoopexport JAVA_HOME=/usr/local/javaexport>export HADOOP_HOME=/home/hadoop/HadoopInstall/hadoopexport HADOOP_CONF_DIR=/home/hadoop/hadoop-confexport PATH=$HADOOP_HOME/bin:$PATHcd $HADOOP_HOME/conf/mkdir /home/hadoop/hadoop-confcp hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml masters slaves /home/hadoop/hadoop-confvi $HADOOP_HOME/hadoop-conf/hadoop-env.sh# The java implementation to use. Required. --修改成你自己jdk安装的目录export JAVA_HOME=/usr/local/java export HADOOP_CLASSPATH=.:/usr/local/java/lib# The maximum amount of heap to use, in MB. Default is 1000.--根据你的内存大小调整export HADOOP_HEAPSIZE=200vi /home/hadoop/.bashrcexport JAVA_HOME=/usr/local/javaexport>export HADOOP_HOME=/home/hadoop/HadoopInstall/hadoopexport HADOOP_CONF_DIR=/home/hadoop/hadoop-confexport PATH=$HADOOP_HOME/bin:$PATH配置namenode#vi $HADOOP_CONF_DIR/slaves192.168.13.108192.168.13.110#vi $HADOOP_CONF_DIR/core-site.xml  fs.default.namehdfs://192.168.13.100:9000#vi $HADOOP_CONF_DIR/hdfs-site.xml  dfs.replication3Default block replication.The actual number of replications can be specified when the file is created.The default is used if replication is not specified in create time.#vi $HADOOP_CONF_DIR/mapred-site.xml  mapred.job.tracker192.168.13.100:11000~在slave上的配置文件如下(hdfs-site.xml不需要配置)：[root@test12 conf]# cat core-site.xmlfs.default.namehdfs://namenode:9000[root@test12 conf]# cat mapred-site.xmlmapred.job.trackernamenode:11000启动export PATH=$HADOOP_HOME/bin:$PATHhadoop namenode -formatstart-all.sh停止stop-all.sh在hdfs上创建danchentest文件夹，上传文件到此目录下$HADOOP_HOME/bin/hadoop fs -mkdir danchentest$HADOOP_HOME/bin/hadoop fs -put $HADOOP_HOME/README.txt danchentestcd $HADOOP_HOMEhadoop jar hadoop-0.20.1-examples.jar wordcount /user/hadoop/danchentest/README.txt output109/12/21 18:31:44 INFO input.FileInputFormat: Total input paths to process : 109/12/21 18:31:45 INFO mapred.JobClient: Running job: job_200912211824_000209/12/21 18:31:46 INFO mapred.JobClient: map 0% reduce 0%09/12/21 18:31:53 INFO mapred.JobClient: map 100% reduce 0%09/12/21 18:32:05 INFO mapred.JobClient: map 100% reduce 100%09/12/21 18:32:07 INFO mapred.JobClient: Job complete: job_200912211824_000209/12/21 18:32:07 INFO mapred.JobClient: Counters: 1709/12/21 18:32:07 INFO mapred.JobClient: Job Counters09/12/21 18:32:07 INFO mapred.JobClient: Launched reduce tasks=1查看输出结果文件，这个文件在hdfs上[root@test11 hadoop]# hadoop fs -ls output1Found 2 itemsdrwxr-xr-x - root supergroup 0 2009-09-30 16:01 /user/root/output1/_logs-rw-r--r-- 3 root supergroup 1306 2009-09-30 16:01 /user/root/output1/part-r-00000[root@test11 hadoop]# hadoop fs -cat output1/part-r-00000(BIS), 1(ECCN) 1查看hdfs运行状态,可以通过web界面来访问http://192.168.13.100:50070/dfshealth.jsp;查看map-reduce信息，可以通过web界面来访问http://192.168.13.100:50030/jobtracker.jsp;下面是直接命令行看到的结果。出现08/01/25 16:31:40 INFO ipc.Client: Retrying connect to server: foo.bar.com/1.1.1.1:53567. Already tried 1 time(s).的原因是没有格式化:hadoop namenode -format