hadoop2.7环境的编译安装
0.环境说明:编译所用的操作系统为:
# cat /etc/redhat-release
CentOS> hadoop的版本为2.7.1
1.安装依赖软件包:
yum install svn autoconf automake libtool cmake ncurses-devel openssl-devel gcc*
2.配置java和maven环境:
wget wget http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz?AuthParam=1443446776_174368b9ab1a6a92468aba5cd4d092d0
tar -zxvf jdk-8u60-linux-x64.tar.gz -C /usr/local
cd /usr/local
ln -s jdk1.8.0_60 jdk
echo 'export JAVA_HOME=/usr/local/jdk' >>/etc/profile;
echo 'export PATH=$JAVA_HOME/bin:$PATH'>>/etc/profile;
wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz
tar -zxvf apache-maven-3.3.3-bin.tar.gz -C /usr/local
cd /usr/local
ln -s apache-maven-3.3.3 maven
echo 'export PATH=/usr/local/maven/bin:$PATH' >/etc/profile.d/maven.sh;
3.下载并安装protobuf(必须使用2.5版本)
wget https://codeload.github.com/google/protobuf/zip/v2.5.0
unzip protobuf-2.5.0.zip
wget http://googletest.googlecode.com/files/gtest-1.5.0.tar.bz2
tar -jxvf gtest-1.5.0.tar.bz2
mv gtest-1.5.0 ./protobuf-2.5.0/gtest
./autogen.sh
./configure
make
make check
make install
which protoc
# which protoc
/usr/local/bin/protoc
4.下载并安装ant:
wget http://mirrors.cnnic.cn/apache//ant/binaries/apache-ant-1.9.6-bin.zip
unzip apache-ant-1.9.6-bin.zip
mv apache-ant-1.9.6 /usr/local/ant
echo 'export PATH=/usr/local/ant/bin:$PATH' >/etc/profile.d/ant.sh
5.编译hadoop:
tar -zxvf tar -zxvf hadoop-2.7.1-src.tar.gz
mvn package -Pdist,native -DskipTests -Dtar
6.故障处理:
第一次编译故障:
Failed to execute goal on project hadoop-auth: Could not resolve dependencies for project org.apache.hadoop:hadoop-auth:jar:2.7.1: The following artifacts could not be resolved: org.mockito:mockito-all:jar:1.8.5, org.mortbay.jetty:jetty-util:jar:6.1.26, org.mortbay.jetty:jetty:jar:6.1.26, org.apache.tomcat.embed:tomcat-embed-core:jar:7.0.55, org.apache.httpcomponents:httpclient:jar:4.2.5, org.apache.zookeeper:zookeeper:jar:3.4.6: Could not transfer artifact org.mockito:mockito-all:jar:1.8.5 from/to central (https://repo.maven.apache.org/maven2): GET request of: org/mockito/mockito-all/1.8.5/mockito-all-1.8.5.jar from central failed: SSL peer shut down incorrectly ->
To see the full stack trace of the errors, re-run Maven with the -e switch.
Re-run Maven using the -X switch to enable full debug logging.
For more information about the errors and possible solutions, please read the following articles:
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
After correcting the problems, you can resume the build with the command
mvn-rf :hadoop-auth
解决办法:
这种情况很常见，这是因为插件没有下载完毕造成的。多执行几次下面命令就可以了
mvn package -Pdist,native -DskipTests -Dtar
第二次编译故障:
Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.1:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 3.0.0', expected version is '2.5.0' ->
protobuf版本过新,需要使用2.5的版本;
7.编译成功的日志:
Apache Hadoop Main ................................. SUCCESS
Apache Hadoop Project POM .......................... SUCCESS
Apache Hadoop Annotations .......................... SUCCESS [ 10.274 s]
Apache Hadoop Assemblies ........................... SUCCESS
Apache Hadoop Project Dist POM ..................... SUCCESS
Apache Hadoop Maven Plugins ........................ SUCCESS [ 11.000 s]
Apache Hadoop MiniKDC .............................. SUCCESS
Apache Hadoop Auth ................................. SUCCESS
Apache Hadoop Auth Examples ........................ SUCCESS
Apache Hadoop Common ............................... SUCCESS
Apache Hadoop NFS .................................. SUCCESS [ 20.029 s]
Apache Hadoop KMS .................................. SUCCESS [ 21.350 s]
Apache Hadoop Common Project ....................... SUCCESS
Apache Hadoop HDFS ................................. SUCCESS
Apache Hadoop HttpFS ............................... SUCCESS
Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 46.255 s]
Apache Hadoop HDFS-NFS ............................. SUCCESS [ 21.495 s]
Apache Hadoop HDFS Project ......................... SUCCESS
hadoop-yarn ........................................ SUCCESS
hadoop-yarn-api .................................... SUCCESS
hadoop-yarn-common ................................. SUCCESS
hadoop-yarn-server ................................. SUCCESS
hadoop-yarn-server-common .......................... SUCCESS [ 35.037 s]
hadoop-yarn-server-nodemanager ..................... SUCCESS [ 44.224 s]
hadoop-yarn-server-web-proxy ....................... SUCCESS
hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 17.461 s]
hadoop-yarn-server-resourcemanager ................. SUCCESS [ 46.435 s]
hadoop-yarn-server-tests ........................... SUCCESS [ 10.698 s]
hadoop-yarn-client ................................. SUCCESS
hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 10.343 s]
hadoop-yarn-applications ........................... SUCCESS
hadoop-yarn-applications-distributedshell .......... SUCCESS
hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS
hadoop-yarn-site ................................... SUCCESS
hadoop-yarn-registry ............................... SUCCESS [ 12.317 s]
hadoop-yarn-project ................................ SUCCESS [ 18.781 s]
hadoop-mapreduce-client ............................ SUCCESS
hadoop-mapreduce-client-core ....................... SUCCESS [ 46.350 s]
hadoop-mapreduce-client-common ..................... SUCCESS [ 34.772 s]
hadoop-mapreduce-client-shuffle .................... SUCCESS
hadoop-mapreduce-client-app ........................ SUCCESS [ 22.440 s]
hadoop-mapreduce-client-hs ......................... SUCCESS [ 12.865 s]
hadoop-mapreduce-client-jobclient .................. SUCCESS
hadoop-mapreduce-client-hs-plugins ................. SUCCESS
Apache Hadoop MapReduce Examples ................... SUCCESS
hadoop-mapreduce ................................... SUCCESS [ 12.782 s]
Apache Hadoop MapReduce Streaming .................. SUCCESS [ 24.680 s]
Apache Hadoop Distributed Copy ..................... SUCCESS [ 50.965 s]
Apache Hadoop Archives ............................. SUCCESS
Apache Hadoop Rumen ................................ SUCCESS [ 12.928 s]
Apache Hadoop Gridmix .............................. SUCCESS
Apache Hadoop Data Join ............................ SUCCESS
Apache Hadoop Ant Tasks ............................ SUCCESS
Apache Hadoop Extras ............................... SUCCESS
Apache Hadoop Pipes ................................ SUCCESS [ 31.548 s]
Apache Hadoop OpenStack support .................... SUCCESS [ 10.084 s]
Apache Hadoop Amazon Web Services support .......... SUCCESS
Apache Hadoop Azure support ........................ SUCCESS [ 36.126 s]
Apache Hadoop Client ............................... SUCCESS [ 24.463 s]
Apache Hadoop Mini-Cluster ......................... SUCCESS
Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 12.506 s]
Apache Hadoop Tools Dist ........................... SUCCESS [ 34.475 s]
Apache Hadoop Tools ................................ SUCCESS
Apache Hadoop Distribution ......................... SUCCESS
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 01:12 h
Finished at: 2015-10-03T03:54:29+08:00
Final Memory: 91M/237M
------------------------------------------------------------------------
#
8.检查生成包:
cd /tmp/hadoop-2.7.1-src/hadoop-dist/target;
# ls -ld hadoop*
drwxr-xr-x 9 root root 4096 10月3 03:51 hadoop-2.7.1
-rw-r--r-- 1 root root 194796372 10月3 03:52 hadoop-2.7.1.tar.gz
-rw-r--r-- 1 root root 2823 10月3 03:52 hadoop-dist-2.7.1.jar
-rw-r--r-- 1 root root 390430395 10月3 03:54 hadoop-dist-2.7.1-javadoc.jar
9.至此编译工作顺利结束;
页:
[1]