oozie：hadoop中的工作流引擎

zhangsanfeng88

一、maven安装
　　1) 从网站上下载Linux版本的Maven安装包；
　　apache-maven-3.0.5-bin.tar.gz下载
　　具体解压目录在 /home/maven/
　　========================================
　　2) 解压apache-maven-3.0.5-bin.tar.gz进行安装；



tar -xzvf apache-maven-3.0.5-bin.tar.gz

　　
　　3) 设置maven环境变量；



vim /etc/profile或者vim /etc/environment

　　在末尾处增加如下内容：



#maven envirment
export M2_HOME=/home/maven
export M2=$M2_HOME/bin
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$M2:$PATH

　　4) 使配置文件生效



source /etc/profile
　　5)验证maven是否安装成功。



root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /home/maven
Java version: 1.6.0_43, vendor: Sun Microsystems Inc.
Java home: /usr/lib/jvm/jdk1.6.0_43/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-24-server", arch: "amd64", family: "unix"

　　
　　二、oozie
　　1、Build



bin/mkdistro.sh -DskipTests

　　成功build后,会在目录里生成一个distribution.

hadoop安装时创建的hadoop组和hadoop用户对oozie目录是没有权限，所以最好一开始创建oozie组和oozie用户，同时赋予权限，不然会报错.偷懒用了root用户。　
　　PS：后来发现，可以单独拷贝/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2下的oozie-3.3.2-distro.tar.gz包到用户自定义目录进行解压使用，除此之外的所有文件可以全部删除。
　　2、安装
　　根据官方文档在distibution的目录下（/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2）创建libext，因为装的hadoop版本是1.0.3,所以拷贝$HADOOP_HOME/lib下的所有jar包到libext下，同时拷贝ext-2.2.zip到该目录下。
还要拷贝$HADOOP_HOME下的所有jar包到libext.　
　　因为oozie默认并不支持1.0.3版，所以对/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin文件夹下的addtowar.sh文件进行修改：



elif [ "${version}" = "0.20.200" ]; then
#List is separated by ":"
hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"
后边增加如下内容：
elif [ "${version}" = "1.0.3" ]; then
#List is separated by ":"
hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"

　　
　　运行oozie-setup.sh创建oozie.war



root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-setup.sh -hadoop 1.0.3 /home/hadoop -extjs /home/oozie/ext-2.2.zip


　　执行成功后会出现如下界面：
　　
　　拷贝oozie.war到tomcat部署目录下



root@ubuntu:/home/oozie/webapp/src/main/webapp# cp /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie.war ./

　　修改配置文件：　　




oozie.service.JPAService.create.db.schema
true

Creates Oozie DB.
If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.
If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.



　　创建Oozie DB



cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin  
./ooziedb.sh create -sqlfile oozie.sql -run

　　结果



DONE
Check DB schema does not exist
DONE
Check OOZIE_SYS table does not exist
DONE
Create SQL schema
DONE
DONE
Create OOZIE_SYS table
DONE
Oozie DB has been created for Oozie version '3.3.1'

　　运行oozie



cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin
./oozie-start.sh　　
　　开启服务：
　　



bin/oozie-run.sh　

　　运行结果　　



root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-run.sh
WARN: Use of this script is deprecated; use 'oozied.sh run' instead
Setting OOZIE_HOME:          /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2
Setting OOZIE_CONFIG:        /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf
Sourcing:                    /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-env.sh
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Setting OOZIE_CONFIG_FILE:   oozie-site.xml
Setting OOZIE_DATA:          /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data
Setting OOZIE_LOG:           /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs
Setting OOZIE_LOG4J_FILE:    oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD:  10
Setting OOZIE_HTTP_HOSTNAME: dm4
Setting OOZIE_HTTP_PORT:     11000
Setting OOZIE_ADMIN_PORT:     11001
Setting OOZIE_HTTPS_PORT:     11443
Setting OOZIE_BASE_URL:      http://dm4:11000/oozie
Setting CATALINA_BASE:       /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE:     /root/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS:     password
Setting CATALINA_OUT:        /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/catalina.out
Setting CATALINA_PID:        /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
Using   CATALINA_OPTS:        -Xmx1024m -Dderby.stream.error.file=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/derby.log
Adding to CATALINA_OPTS:     -Doozie.home.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2 -Doozie.config.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf -Doozie.log.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs -Doozie.data.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=dm4 -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://dm4:11000/oozie -Doozie.https.keystore.file=/root/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=
Using CATALINA_BASE:   /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Using CATALINA_HOME:   /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Using CATALINA_TMPDIR: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp
Using JRE_HOME:        /usr/lib/jvm/jdk1.6.0_43/jre
Using CLASSPATH:       /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/bin/bootstrap.jar
Using CATALINA_PID:    /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:
Jun 2, 2013 3:06:46 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-11000
Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 338 ms
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.36
Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor oozie.xml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-simple-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
log4j:WARN No appenders could be found for logger (org.apache.jasper.compiler.JspRuntimeContext).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Jun 2, 2013 3:06:50 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-11000
Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3976 ms

　　界面：
　　和例子界面类似，没有内容。
　　
　　
　　案例：
　　将/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2中的oozie-examples.tar.gz解压，出现examples文件夹
　　修改



/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples/map-reduce/job.properties文件为如下形式：
nameNode=hdfs://211.80.198.148:9000
jobTracker=211.80.198.148:9001
queueName=default
examplesRoot=examples
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce


　　
　　将文件夹上传到集群



root@dm4:/home/hadoop# bin/hadoop fs -copyFromLocal /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples /user/root

　　
　　运行命令



root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
　　
运行结果：



job: 0000000-130602150647008-oozie-root-W

　　不上传会出现如下错误：



root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
Error: E0504 : E0504: App directory [hdfs://211.80.198.148:9000/user/root/examples/apps/map-reduce] does not exist

　　
　　
　　结果

　　PS：中间出现的一些错误
　　



ERROR: Oozie could not be started
REASON: java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils

　　将hadoop的jar包（主目录下和lib目录下）拷贝到如下位置



/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib

　　之后，运行正常。
　　
　　参考：
1、http://angelosun.iteye.com/blog/1835881
　　2、http://practicalcloudcomputing.com/post/26337621577/installing-and-running-apache-oozie-3-2-x-and-possibly
　　3、http://wsysisibeibei.blog.163.com/blog/static/16536806120126233155505/
　　
　　转载请保留：http://www.iyunv.com/Deron/archive/2013/06/02/3111178.html