坑爹的Apache hbase 64位机安装Snappy终于成功了
坑爹的网上copy来copy去，都没有一个执行成功的。还是自己记录下来算了。Apache hadoop2.2.0 hbase 0.96 ,CentOS release 6.5 (Final)64位
1.安装基本tool
yum install gcc c++, autoconf, automake, libtool, Java 6, JAVA_HOME set, Maven 3,svn
yum Error: Cannot retrieve repository metadata (repomd.xml) for repository: xxxxx
sodu vim /etc/yum.repos.d/xxxxx.repo
将项中的enabled=1改为enabled=0
解决yum源的问题。
2.安装Snappy
下载snappy
wget http://snappy.googlecode.com/files/snappy-1.1.1.tar.gz
然后解压后，执行三步骤：
./configure
make
sudo make install
默认安装路径：/usr/local/lib下面
检查安装是否成功
ls /usr/local/lib/libsn*
3.安装hadoop-snappy
3.1下载hadoop-snappy源码
svn checkout http://hadoop-snappy.googlecode.com/svn/trunk/ hadoop-snappy
3.2.安装hadoop-snappy
cd hadoop-snappy
mvn package
4.hadooo中部署snappy
解压hadoop-snappy-0.0.1-SNAPSHOT.tar.gz文件，会生成hadoop-snappy-0.0.1-SNAPSHOT目录，拷贝这个目录下相关文件到$HADOOP_HOME/lib/native/Linux-amd64-64
cp -r /hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/Linux-amd64-64
将target目录下的hadoop-snappy-0.0.1-SNAPSHOT.jar拷贝到$HADOOP_HOME/lib/目录下。
修改三个文件：
hadoop-env.sh，增加内容如下：
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/
修改core-site.xml文件，增加红色字体部分
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec,
org.apache.hadoop.io.compress.SnappyCodec
</value>
</property>
5.往HBase中使用压缩方式
当hadoop的snappy配置成功后，配置hbase就很简单了，两个步骤：
第一步骤复制相关jar包
cp -r $HADOOP_HOME/lib/native/Linux-amd64-64/* $HBASE_HOME/lib/native/Linux-amd64-64/*
这里需要注意下，有些版本在安装过程中，没有这个Linux-amd64-64这个目录，需要手工创建下。
第二步骤配置hbase-env.sh环境变量
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/
export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/
6、重启Hadoop、HBase 检查安装是否成功
cd $HBASE_HOME/bin
./hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/testfile snappy
结果：Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
坑爹的Apache官网提供的是32位编译的，在64位服务器上会有问题。官方竟然没有提供64位版本，要使用得自己编译。
7.编译hadoop2.2.0
7.1 yum install cmake zlib1g-dev pkg-config libssl-dev
7.2 安装protobuf-2.5.0
很多博客的protobuf的安装都是shit.不知道他们实践过没有，老是copy来copy去。
下载protobuf-2.5.0.tar.gz，解压。
sudo vim /etc/profile
#protobuf
export PROTOC_HOME=/opt/protobuf-2.5.0
export PATH=$PATH:$PROTOC_HOME/src
source /etc/profile
$protoc --version
libprotoc.2.5.0
ok就这样。根本不需要什么configure --prefix,make,make install这么麻烦，也不成功。
7.3 下载hadoop2.2.0源码
[*]Download Hadoop sources.
Patch sources:
cd hadoop-2.2.0-src
wget https://issues.apache.org/jira/secure/attachment/12614482/HADOOP-10110.patch
patch -p0 < HADOOP-10110.patch
maven国外服务器可能连不上，maven配置一下国内镜像，在maven目录下，conf/settings.xml,在<mirrors></mirros>里添加，原本的不要动
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
同样，在<profiles></profiles>内新添加
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.4</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
7.4 编译mvn package -Pdist,native -DskipTests -Dtar -rf :hadoop-common
编译完成了后，cd hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0
file 一下native下面的so文件
将 native/*再cp 到$hadoop_home/bin的各个data node的native/* 和native/Linux-amd64-64下。
重新运行测试，结果
Exception in thread "main" java.lang.RuntimeException:native snappy library not available: this version of libhadoop was built without snappy support.
7.5 mvn package -Pdist,native -DskipTests -Dtar -rf :hadoop-common -Drequire.snappy -X
重新编译，再scp一把 native/*下面的一切
再运行测试OK.
hbase(main):003:0> describe 'mytable'
DESCRIPTION ENABLED
'mytable', {NAME => 'colfam', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPR true
ESSION => 'LZO', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false
', BLOCKCACHE => 'true'}
1 row(s) in 0.0600 seconds
hbase(main):004:0> create 'mytable2', {NAME=>'colfam', COMPRESSION=>'snappy'}
0 row(s) in 0.4430 seconds
=> Hbase::Table - mytable2
hbase(main):005:0> describe 'mytable2'
DESCRIPTION ENABLED
'mytable2', {NAME => 'colfam', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMP true
RESSION => 'SNAPPY', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'f
alse', BLOCKCACHE => 'true'}
1 row(s) in 0.0550 seconds
页:
[1]