Install oozie-3.3.2 on Hadoop 1.1.1

61581229

After a few hours tweaking and googling, I managed to install apache oozie 3.3.2 on Hadoop 1.1.1.
The document provided in apache oozie 3.3.2 is not very clear. After some googling, I found this blog is a great guide for you install oozie 3.3.1 on Hadoop 0.23.0.
http://srikanthayalasomayajulu.blogspot.sg/2013/02/apache-oozie-331-installation-on-apache.html
If you want to install oozie on MRv2/YARN, please follow the blog post.
 
Here we go, we're going to install oozie 3.3.2 on Hadoop 1.1.1, which is offically supported version.
Apache oozie 3.3.2 official quick start page: http://oozie.apache.org/docs/3.3.2/DG_QuickStart.html
 
My local environment:

• Apache Hadoop 1.1.1 pseudo setup
  
• Apache oozie 3.3.2, the latest release
  
• Java 1.7 u25
  
• Apache Maven 3.0.4
  

Oozie server installation:

• Download apache oozie 3.3.2 via: http://www.apache.org/dyn/closer.cgi/oozie/3.3.2
  
• Unpack the oozie-3.3.2.tar.gz under development directory: ~/dev/oozie-3.3.2
  
• The following two properties are required in hadoop core-site.xml:
  
• 
  <!-- OOZIE -->
  <property>
  <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
  <value>[OOZIE_SERVER_HOSTNAME]</value>
  </property>
  <property>
  <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
  <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
  </property>
   Replaced [OOZIE_SERVER_USER] with gsun , [OOZIE_SERVER_HOSTNAME] with * and [OOZIE_GROUPS_THAT_ALLOW_IMPERSONATION] with *  . TIP: in this blog post, the $OOZIE_HOME refers to ~/dev/oozie-3.3.2
  
• Download extjs-2.2.zip, don't unzip it right now.
  
• Building a distro
  
• 
  cd $OOZIE_HOME
  ./bin/mkdistro.sh -DskipTests
   It will take a while, go get a coffee, you'll be fine.
  
• If the build successfully, we could go to the next step. In $OOZIE_HOME a directory called  *hadooplibs/* directory will be created containing the Hadoop JARs for the versions of Hadoop that the Oozie distribution supports.
  MacBookPro:oozie-3.3.2 gsun$ cd hadooplibs/
  MacBookPro:hadooplibs gsun$ ls
  hadoop-1/        hadoop-3/        hadoop-distcp-2/ hadoop-test-1/   hadoop-test-3/   target/
  hadoop-2/        hadoop-distcp-1/ hadoop-distcp-3/ hadoop-test-2/   pom.xml
  MacBookPro:hadooplibs gsun$ tree hadoop-1/
  hadoop-1/
  ├── pom.xml
  └── target
  ├── archive-tmp
  ├── hadooplibs
  │   └── hadooplib-1.1.1.oozie-3.3.2
  │       ├── commons-beanutils-1.7.0.jar
  │       ├── commons-beanutils-core-1.8.0.jar
  │       ├── commons-codec-1.4.jar
  │       ├── commons-collections-3.2.1.jar
  │       ├── commons-configuration-1.6.jar
  │       ├── commons-digester-1.8.jar
  │       ├── commons-el-1.0.jar
  │       ├── commons-io-2.1.jar
  │       ├── commons-lang-2.4.jar
  │       ├── commons-logging-1.1.jar
  │       ├── commons-math-2.1.jar
  │       ├── commons-net-1.4.1.jar
  │       ├── hadoop-client-1.1.1.jar
  │       ├── hadoop-core-1.1.1.jar
  │       ├── hsqldb-1.8.0.7.jar
  │       ├── jackson-core-asl-1.8.8.jar
  │       ├── jackson-mapper-asl-1.8.8.jar
  │       ├── log4j-1.2.16.jar
  │       ├── oro-2.0.8.jar
  │       └── xmlenc-0.52.jar
  ├── maven-archiver
  │   └── pom.properties
  └── oozie-hadoop-1.1.1.oozie-3.3.2.jar
  5 directories, 23 files
  MacBookPro:hadooplibs gsun$
   
  
• Create a directory named libext/ in $OOZIE_HOME, copy all of jars inside hadoop-1 into $OOZIE_HOME/libext/
  
• Building with ExtJS library for Oozie Web Console. Copy the extjs-2.2.zip into $OOZIE_HOME/webapp/src/main/webapp, then unzip it.
  
• Run oozie-setup.sh to create an oozie.war file.   
  > cd $OOZIE_HOME
  > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie-setup.sh  -extjs $OOZIE_HOME/webapp/src/main/webapp/ext-2.2.zip
   If the build success, go to the next step, or refer to apache Oozie official docs for oozie-setup.sh usage.
  
• Copy the newly minted oozie.war file to your Tomcat deployment directory.
  > cd $OOZIE_HOME
  > vi ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-site.xml
   Change the property below to true as shown.
  <property>
  <name>oozie.service.JPAService.create.db.schema</name>
  <value>true</value>
  <description>
  Creates Oozie DB.
  If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.
  If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.
  </description>
  </property>
   
  
• Run below command:
  > cd $OOZIE_HOME
  > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/ooziedb.sh create -sqlfile oozie.sql -run
  if the command is successfully run, the SQL has been created successfully.
  Start the oozie server.
  

> cd $OOZIE_HOME
./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozied.sh run

 From quick start in Oozie official site,  we know that all Oozie server scripts (=oozie-setup.sh=, oozied.sh , oozie-start.sh , oozie-run.sh and oozie-stop.sh ) run only under the Unix user that owns the Oozie installation directory, if necessary use sudo -u OOZIE_USER when invoking the scripts.As of Oozie 3.3.2, use of oozie-start.sh , oozie-run.sh , and oozie-stop.sh has been deprecated and will print a warning. The oozied.sh script should be used instead; passing it start , run , or stop as an argument will perform the behaviors of oozie-start.sh , oozie-run.sh , and oozie-stop.sh respectively.Check the Oozie log file  ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/oozie.log to ensure Oozie started properly.

 

Using the Oozie command line tool check the status of Oozie:

 

$ ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie admin -oozie http://[ip address of namenode]:11000/oozie -statusIn my case, the Oozie server can not start due to below exception:

java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils

 You have to copy hadoop-core.jar and commons-configuration.jar under $HADOOP_HOME/lib into $OOZIE_HOME/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib . You should see the oozie server up and running. Enjoy oozie!

Another problem I met during development is that oozie thrown below exception after job submitted:


Exception in thread "pool-2-thread-25" java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException
at org.apache.oozie.service.KerberosHadoopAccessorService$1.run(KerberosHadoopAccessorService.java:129)
at org.apache.oozie.service.KerberosHadoopAccessorService$1.run(KerberosHadoopAccessorService.java:127)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.oozie.service.KerberosHadoopAccessorService.createJobClient(KerberosHadoopAccessorService.java:127)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:789)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:534)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:746)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:193)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:53)
at org.apache.oozie.command.XCommand.call(XCommand.java:257)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:163)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.JsonMappingException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
... 16 more


To solve this problem, you have to copy jackson-core-asl.jar 和 jackson-mapper-asl.jar from $HADOOP_HOME/lib to $OOZIE_HOME/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib.