Install hadoop-lzo-lib for Hadoop v1.1.1 on Mac OS X with Mountain Lion

晋君丽

Here is a quick guide to install hadoop lzo library, the install process is pretty easy actually.　　

• 
  [size=1em]Install LZO with HomeBrew, [size=1em]LZO is a wonderful compression scheme to use with Hadoop. [size=1em]HomeBrew is prerequisite, visit [size=1em]https://github.com/mxcl/homebrew to see how to install HomeBrew.
  

brew install lzo
  

• Clone the hadoop-lzo source code to local:
  

git clone https://github.com/twitter/hadoop-lzo.git ~/code/hadoop-lzo && cd ~/code/hadoop-lzo
 

• Build hadoop-lzo to generate hadoop-lzo-lib as well as hadoop lzo native library, find the built artifacts in the target directory.
  

C_INCLUDE_PATH=/usr/local/Cellar/lzo/2.06/include \
LIBRARY_PATH=/usr/local/Cellar/lzo/2.06/lib \
mvn clean test package
 Waiting for build complete. If something wrong, refer to https://github.com/twitter/hadoop-lzo for troubleshoting.

• Copy the built artifacts into hadoop lib directory or native library directory respectively.
  

 

• Once the libs are built and installed, you may want to add them to the class paths and library paths. That is, in hadoop-env.sh, set 
  

export HADOOP_CLASSPATH=$HADOOP_HOME/lib/hadoop-lzo-xxxx.jar
export JAVA_LIBRARY_PATH=/path/to/hadoop-lzo-native-libs:/path/to/standard-hadoop-native-libs
 

• Note that there seems to be a bug in /path/to/hadoop/bin/hadoop; (refer to https://issues.apache.org/jira/browse/HADOOP-6453) . Comment out the line
  

    JAVA_LIBRARY_PATH=''
　　
　　See https://github.com/twitter/hadoop-lzo for further instruction if you have any question or leave a comment.