日志系统系列flume-ng+kafka+storm+HDFS搭建实验

amy_888

　　
最近公司要求搞个日志系统，以前接触不多，最近看了好多资料什么flume flume-ng kafka storm hdfs es 等等，感觉其实要做个小的直接kafka或者flume-ng就可以(现在已经代替了flume)，看了N多大神的文章比如这个《实时系统搭建》就很不错，但是对于我这种新手来说不一定合适，所以自己就准备边搞边记录了！
　　首先我的技术选型就是flume-ng kafka  storm hdfs 其中每个技术我都会详细写出步骤，省的到时候自己忘记了，原型就是根据Jonathan-Wei大神的文章来的，上边已经说了就不再重复！
　　 好了下边就来搞搞flume-ng，大家应该都知道与flume的不同之处吧！不知道的小伙伴也不要紧请看转载的文章flume的革命好了，下面我们就来看下flume的搭建
　　flume-ng的东西在整理下
　　===================================================================================================================================
　　hadoop hbase 单机版搭建
　　后面需要安装集群，整合到hadoop，所以注意选择与hadoop对应的版本：


　　
　　Table 2.1. Hadoop version support matrix


HBase-0.92.x
HBase-0.94.x
HBase-0.96

Hadoop-0.20.205
S
X
X
Hadoop-0.22.x
S
X
X
Hadoop-1.0.x
S
S
S
Hadoop-1.1.x
NT
S
S
Hadoop-0.23.x
X
S
NT
Hadoop-2.x
X
S
S　　





S = supported and tested,支持
X = not supported,不支持
NT = not tested enough.可以运行但测试不充分

由于 HBase 依赖 Hadoop，它配套发布了一个Hadoop jar 文件在它的 lib 下。该套装jar仅用于独立模式。在分布式模式下，Hadoop版本必须和HBase下的版本一致。用你运行的分布式Hadoop版本jar文件替换HBase lib目录下的Hadoop jar文件，以避免版本不匹配问题。确认替换了集群中所有HBase下的jar文件。Hadoop版本不匹配问题有不同表现，但看起来都像挂掉了。


　　

安装：

$ tar xfz hbase-0.90.4.tar.gz

$ cd hbase-0.90.4
我自己使用的是独立的zookeeper 后边连接的是hdfs存储，所以有如下配置
hbase.rootdir 必须和 hadoop的core-sit.xml里面的地址一致




　　

编辑配置文件hbase-site.xml

• vi hbase-site.xml
  
• <configuration>
  
•   <property>
  
•     <name>hbase.rootdir</name>
  
•     <value>hdfs://server01:9000/hbase</value>
  
•     <description>区域服务器使用存储HBase数据库数据的目录</description>
  
•   </property>
  
•   <property>
  
•     <name>hbase.cluster.distributed</name>
  
•     <value>true</value>
  
•     <description>
  
• 指定HBase运行的模式：
  
• false: 单机模式或者为分布式模式                  
  
•                  true: 全分布模式  
  
•     </description>
  
•   </property>
  
•   <property>
  
•     <name>hbase.zookeeper.quorum</name>
  
•     <value>server01,server02,server03</value>
  
•     <description>ZooKeeper集群服务器的位置</description>
  
•   </property>
  
• </configuration>
  

设置环境变量hbase-env.sh

• vi hbase-evn.sh  
  
• export HBASE_MANAGES_ZK=false
  

使用独立的ZooKeeper时需要修改HBASE_MANAGES_ZK&#20540;为false，为不使用默认ZooKeeper实例。





我的环境变量

==============================================================================





# hadoop

HADOOP_HOME=/home/mq/hadoop/hadoop

# hbase

HBASE_HOME=/home/mq/hbase/hbase-0.98.7

# flume

export FLUME_HOME=/home/mq/flume/flume

# sbt

SBT=/home/mq/kafka/sbt

# kafka

KAFKA=/home/mq/kafka/kafka

JAVA_HOME=/usr/java/jdk1.7.0_60

CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$JAVA_HOME/lib/tools.jar:$CLASS_PATH:$HBASE_HOME/bin



=============================================================================================================================

单机启动抛出这样一个错误



[mq@localhost hbase-0.98.7]$ ./bin/start-hbase.sh
2014-10-23 10:46:27,658 FATAL [main] conf.Configuration: error parsing conf file: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
Exception in thread &quot;main&quot; java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)
at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)
at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)
at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)
at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)
Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:687)
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:557)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
... 8 more
starting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.out
localhost: +======================================================================+
localhost: |      Error: JAVA_HOME is not set and Java could not be found         |
localhost: +----------------------------------------------------------------------+
localhost: | Please download the latest Sun JDK from the Sun Java web site        |
localhost: |       > http://java.sun.com/javase/downloads/ <                      |
localhost: |                                                                      |
localhost: | HBase requires Java 1.6 or later.                                    |
localhost: | NOTE: This script will find Sun Java whether you install using the   |
localhost: |       binary or the RPM based installer.                             |
localhost: +======================================================================+


　　

我这很费解，看样子应该是环境变量问题修改修改conf文件下的hbase-env.sh文件中的java_home的环境变量

修改完以后错误变化

[mq@localhost hbase-0.98.7]$ ./bin/start-hbase.sh
2014-10-23 10:59:39,411 FATAL [main] conf.Configuration: error parsing conf file: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
Exception in thread &quot;main&quot; java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)
at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)
at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)
at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)
at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)
Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:687)
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:557)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)
... 8 more
starting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.out
localhost: starting regionserver, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-regionserver-localhost.localdomain.out
localhost: Exception in thread &quot;main&quot; java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。
localhost:      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)
localhost:      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)
localhost:      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)
localhost:      at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)
localhost:      at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)
localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)
localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)
localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)
localhost:      at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2484)</pre><pre name=&quot;code&quot; class=&quot;html&quot; style=&quot;color: rgb(51, 51, 51); font-family: Arial; font-size: 14px; line-height: 26px;&quot;>这个是编码问题，代表你有的配置文件不是utf-8的，仔细检查下吧还有一些问题比如报警告<h3 class=&quot;t&quot; style=&quot;margin: 0px 0px 1px; padding: 0px; list-style: none; font-weight: normal;font-size:14px; line-height: 1.54; color: rgb(51, 51, 51); font-family: arial;&quot;><a target=_blank data-click=&quot;{'F':'378717EA','F1':'9D73F1E4','F2':'4CA6DE6B','F3':'54E5243F','T':'1414034754','y':'BF6CF7B5' }&quot; href=&quot;http://www.baidu.com/link?url=o9-rxdk2Vum7yUDsx_F4fRvvG_VAwahcWTbiEX1l3f-SFwRNFGTSTCbwvFASirhz9bMWR1SQLuPu_IFXNM8hSa&quot; target=&quot;_blank&quot;><span style=&quot;color: rgb(204, 0, 0);&quot;>WARN</span> <span style=&quot;color: rgb(204, 0, 0);&quot;>util.NativeCodeLoader:</span> <span style=&quot;color: rgb(204, 0, 0);&quot;>Unable</span> <span style=&quot;color: rgb(204, 0, 0);&quot;>to</span> <span style=&quot;color: rgb(204, 0, 0);&quot;>load</span> <span style=&quot;color: rgb(204, 0, 0);&quot;>native-hadoop</span> ...</a></h3><span style=&quot;word-wrap: normal; word-break: normal; line-height: 27px; color: rgb(50, 62, 50); font-size: 13.5pt; font-family: 微软雅黑, sans-serif; background-color: rgb(145, 157, 163);&quot;>Apache</span><span style=&quot;word-wrap: normal; word-break: normal; line-height: 27px; color: rgb(50, 62, 50); font-size: 13.5pt; font-family: 微软雅黑, sans-serif; background-color: rgb(145, 157, 163);&quot;>提供的<span style=&quot;word-wrap: normal; word-break: normal;&quot;>hadoop</span>本地库是<span style=&quot;word-wrap: normal; word-break: normal;&quot;>32</span>位的，而在<span style=&quot;word-wrap: normal; word-break: normal;&quot;>64</span>位的服务器上就会有问题，因此需要自己编译<span style=&quot;word-wrap: normal; word-break: normal;&quot;>64</span>位的版本</span>
启动服务 bin/stop-hbase.sh <pre name=&quot;code&quot; class=&quot;plain&quot;>starting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.out
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/mq/hbase/hbase-0.98.7/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/mq/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.


　　


我去掉slf4j-log4j12-1.5.8.jar就解决了问题，没有报错。具体还有待查找　　

下面我们来验证下是否正确了

发现不对劲，居然没反应报错了，看了下日志

java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:202)
at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:152)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:179)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2786)
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1113)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:926)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:460)
at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:258)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:137)
... 7 more

无语的问题啊，继续查吧看来下，貌&#20284;hadoop和hbase版本不合的问题，突然发型hadoop2.5.1没有core，无语了啊　　发现最后还是版本问题，我勒个去啊具体版本可以参考http://www.aboutyun.com/blog-61-62.html
　　换了个同样的版本的2就可以了，然后zookeeper又不能连接2222端口，本来服务器已经有了个zookeeper想用同一个或者再来一个
　　编辑hbase-site.xml 加入地址（都在hosts配置好了）和端口2222


　　<property>

   <name>hbase.zookeeper.quorum</name>

   <value>hd204,hd205,hd206</value>

</property>


　　在编辑regionservers加入映射
　　hd204

hd205

hd206


　　OK 启动hbase
　　http://192.168.1.116:60010/master-status#


　　查看hbase状态，简单的配置就完成了
　　后期会把hadoop&#43;flume-ng的东西补上留作参考，骑士hbase还是不全，有时间在整理了