设为首页 收藏本站

运维网

 找回密码
 立即注册
查看: 1380|回复: 0

[经验分享] [kylin] 部署kylin服务

[复制链接]
xiu12

尚未签到
发表于 2017-3-1 10:13:12 | 显示全部楼层 |阅读模式
一、工具准备



zookeeper3.4.6 （hadoop、hbase 管理工具）
Hadoop.2.7.1
Hbase1.1.4
Kylin1.5.0-HBase1.1.3
Jdk1.7.80
Hive 2.0.0
二、虚拟主机



192.168.200.165 master1
192.168.200.166 master2
192.168.200.167 slave1
192.168.200.168 slave2
三、安装mysql
　　查看是否安装了mysql（master1）



[iyunv@master1 ~]# ps -aux | grep mysql
Mysql 3632 0.0 0.0 115348 1648?SsApr01 0:00 /bin/sh /wdcloud/app/mysql/bin/mysqld_safe
mysql 4519 0.5 19.8 13895940 1591664 ? Sl Apr01 29:55
/wdcloud/app/mysql/bin/mysqld
--basedir=/wdcloud/app/mysql
--datadir=/wdcloud/data/mysql/data
--plugin-dir=/wdcloud/app/mysql/lib/mysql/plugin
--log-error=/wdcloud/data/mysql/data/mysql-error.log
--open-files-limit=20000
--pid-file=/wdcloud/data/mysql/data/localhost.localdomain.pid
--socket=/tmp/mysql.sock
--port=3306
　　查看mysql版本



[iyunv@master1 ~]# mysql --version
mysql  Ver 14.14 Distrib 5.6.29-76.2, for Linux (x86_64) using  6.2
　　登录mysql



[iyunv@master1 ~]# mysql -uroot -p
Enter password:
四、安装jdk
　　查看安装版本



[iyunv@master1 ~]# java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
　　查看安装位置



[iyunv@master1 ~]# which java
/jdk1.7.0_80/bin/java
五、安装zookeeper
　　1.解压缩zookeeper到根目录下，进入目录，创建文件夹data、datalog、logs，配置环境变量



export ZOOKEEPER_HOME=/zookeeper-3.4.6
export PATH=$PATH:$ZOOKEEPER_HOME/bin
　　2.进入conf文件夹，复制zoo_sample.cfg 为zoo.cfg
　　3.修改zoo.cfg，增加红色内容，保存退出：



tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper-3.4.6/data
dataLogDir=/zookeeper-3.4.6/datalog
clientPort=2181
server.0=master1:2888:3888
server.1=master2:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888
　　修改conf下的log4j.properties文件，配置log文件生成位置



zookeeper.log.dir=/zookeeper-3.4.6/logs
zookeeper.log.file=zookeeper.log
zookeeper.tracelog.dir=/zookeeper-3.4.6/logs
zookeeper.tracelog.file=zookeeper_trace.log
　　4.进入data文件夹创建文件“myid”，添加内容 0 保存退出
　　5.分发zookeeper文件夹到各个虚拟主机上的根目录上



scp –r /zookeeper-3.4.6 hadoop@master2:/
scp –r /zookeeper-3.4.6 hadoop@slave1:/
scp –r /zookeeper-3.4.6 hadoop@ slave2:/
　　6.修改每台主机上的myid ，按照顺序 master1 的myid 为0 master2 的myid 1 以此类推。
　　7.启动zookeeper
　　分别进入各个虚拟主机的zookeeper目录下启动zk服务



bin/zkServer.sh start
　　8.分别查询zookeeper的状态



bin/zkServer.sh status


[hadoop@master1 ~]$ zkServer.sh status
JMX enabled by default
Using config: /zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower
　　Leader 是 zookeeper 主机启动了
　　Follower是zookeeper 从机启动了
　　9.停止zookeeper



Bin/zkServer stop

六、hadoop 高可用部署
　　1.解压缩hadoop2.7.1到hadoop家目录下，进入目录，创建文件夹tmp，hdfs/name，hdfs/data
　　2.进入~/hadoop/etc/hadoop，该文件夹包含了hadoop的大部分配置文件。
　　3.修改hadoop各配置文件如下：
　　core-site.xml



<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master1:9000</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop/tmp</value>
<description>A base for other tempory directories</description>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131702</value>
</property>
<property>
<name>fs.checkpoint.period</name>
<value>3600</value>
<description>多长时间记录一次hdfs的镜像，默认一小时</description>
</property>
<property>
<name>fs.checkpoint.size</name>
<value>67108864</value>
<description>一次记录多大的size，默认64M</description>
</property>
</configuration>
　　hadoop.env.sh



#设置JAVA_HOME
export JAVA_HOME=/jdk1.7.0_80
#设置HADOOP_CONF_DIR
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
　　hbase-site.xml



<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master1:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master1,master2,slave1, slave2</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value> /zookeeper-3.4.6 /data</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.coprocessor.user.region.classes</name>
<value>org.apache.hadoop.hbase.coprocessor.AggregateImplementation</value>
</property>
<property>
<name>hbase.regionserver.wal.codec</name>
<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property>
<property>
<name>hbase.master.loadbalancer.class</name>
<value>org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer</value>
</property>
<property>
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
</property>
<property>
<name>phoenix.query.maxServerCacheBytes</name>
<value>1073741824</value>
</property>
<property>
<name>hbase.client.scanner.caching</name>
<value>5000</value>
<description>HBase客户端扫描缓存，对查询性能有很大帮助</description>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>360000000</value>
</property>
</configuration>
　　hdfs-site.xml



<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>4</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master2:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit</name>
<value>false</value>
</property>
</configuration>
　　mapred-site.xml



<configuration>
　　<property>
　　<name>mapreduce.framework.name</name>
　　<value>yarn</value>
</property>
　　<property>
　　<name>mapreduce.jobtracker.http.address</name>
　 <value>NameNode:50030</value>
　　</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master1:19888</value>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
　　新增masters文件，以部署高可用的hadoop，将master2作为Secondary Name Node
　　masters



master2
　　slaves



master1
master2
slave1
slave2
　　yarn-env.sh



export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
export JAVA_HOME=/jdk1.7.0_80
JAVA=$JAVA_HOME/bin/java
JAVA_HEAP_MAX=-Xmx4096m
　　yarn-site.xml



<configuration>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>master1:2181,master2:2181,slave1:2181,slave2:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master1:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
</configuration>
　　4.配置hadoop环境变量



export HADOOP_HOME=/home/hadoop/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
　　
5.查看hadoop是否安装配置成功



[hadoop@master1 ~]$ hadoop version
Hadoop 2.7.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git
-r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
Compiled by jenkins on 2015-06-29T06:04Z
Compiled with protoc 2.5.0
From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
This command was run using
/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar
　　6.分发hadoop文件夹到各个虚拟主机上的hadoop家目录上



scp -r  ~/hadoop/ hadoop@master2:~
scp -r  ~/hadoop/ hadoop@slave1:~
scp -r  ~/hadoop/ hadoop@slave2:~
　　7.重新格式化hdfs系统
　　如果集群刚配置从没启动过，直接执行格式化操作。
　　如果集群已经格式化了后启动过，则先执行删除旧数据的操作后，在执行格式化操作。
　　1）删除旧数据
　　在 hdfs-ste.xml 配置了
　　dfs.name.dir = /home/hadoop/hdfs/name (namenode上存储hdfs名字空间元数据)
　　dfs.data.dir = /home/hadoop/hdfs/data (datanode上数据块的物理存储位置)
　　在core-site.xml中配置了
　　hadoop.tmp.dir = /home/hadoop/hadoop/tmp（namenode上本地的hadoop临时文件夹）
　　将各个集群节点这三个文件夹下面的文件和目录全部删除
　　2）执行格式化命令
　　hadoop namenode -format
　　3）格式化日志



DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
16/04/05 02:02:28 INFO namenode.NameNode: STARTUP_MSG:
/***********************************************************
STARTUP_MSG:   Starting NameNode
STARTUP_MSG:   host = master1/192.168.200.165
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.1
STARTUP_MSG:   classpath =
/home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.7.1-tests.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-nfs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1-tests.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.jar:/home/hadoop/hadoop/contrib/capacity-scheduler/*.jar:/home/hadoop/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git
-r 15ecc87ccf4a0228f35af08fc56de536e6ce657a; compiled by 'jenkins' on 2015-06-29T06:04Z
STARTUP_MSG:   java = 1.7.0_80
************************************************************/
16/04/05 02:02:28 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/04/05 02:02:28 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-070fc765-1b22-4453-83ba-7635ea906e1d
16/04/05 02:02:29 INFO namenode.FSNamesystem: No KeyProvider found.
16/04/05 02:02:29 INFO namenode.FSNamesystem: fsLock is fair:true
16/04/05 02:02:29 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/04/05 02:02:29 INFO blockmanagement.DatanodeManager:
dfs.namenode.datanode.registration.ip-hostname-check=true
16/04/05 02:02:29 INFO blockmanagement.BlockManager:
dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/04/05 02:02:29 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Apr 05 02:02:29
16/04/05 02:02:29 INFO util.GSet: Computing capacity for map BlocksMap
16/04/05 02:02:29 INFO util.GSet: VM type= 64-bit
16/04/05 02:02:29 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
16/04/05 02:02:29 INFO util.GSet: capacity = 2^21 = 2097152 entries
16/04/05 02:02:30 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/04/05 02:02:30 INFO blockmanagement.BlockManager: defaultReplication= 3
16/04/05 02:02:30 INFO blockmanagement.BlockManager: maxReplication= 512
16/04/05 02:02:30 INFO blockmanagement.BlockManager: minReplication= 1
16/04/05 02:02:30 INFO blockmanagement.BlockManager: maxReplicationStreams= 2
16/04/05 02:02:30 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks= false
16/04/05 02:02:30 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/04/05 02:02:30 INFO blockmanagement.BlockManager: encryptDataTransfer= false
16/04/05 02:02:30 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
16/04/05 02:02:30 INFO namenode.FSNamesystem: fsOwner= hadoop (auth:SIMPLE)
16/04/05 02:02:30 INFO namenode.FSNamesystem: supergroup= supergroup
16/04/05 02:02:30 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/04/05 02:02:30 INFO namenode.FSNamesystem: HA Enabled: false
16/04/05 02:02:30 INFO namenode.FSNamesystem: Append Enabled: true
16/04/05 02:02:30 INFO util.GSet: Computing capacity for map INodeMap
16/04/05 02:02:30 INFO util.GSet: VM type= 64-bit
16/04/05 02:02:30 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
16/04/05 02:02:30 INFO util.GSet: capacity= 2^20 = 1048576 entries
16/04/05 02:02:30 INFO namenode.FSDirectory: ACLs enabled? false
16/04/05 02:02:30 INFO namenode.FSDirectory: XAttrs enabled? true
16/04/05 02:02:30 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
16/04/05 02:02:30 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/04/05 02:02:30 INFO util.GSet: Computing capacity for map cachedBlocks
16/04/05 02:02:30 INFO util.GSet: VM type= 64-bit
16/04/05 02:02:30 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
16/04/05 02:02:30 INFO util.GSet: capacity= 2^18 = 262144 entries
16/04/05 02:02:30 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/04/05 02:02:30 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/04/05 02:02:30 INFOnamenode.FSNamesystem: dfs.namenode.safemode.extension= 30000
16/04/05 02:02:30 INFO metrics.TopMetrics: NNTop conf:
dfs.namenode.top.window.num.buckets = 10
16/04/05 02:02:30 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
16/04/05 02:02:30 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
16/04/05 02:02:30 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/04/05 02:02:30 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/04/05 02:02:30 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/04/05 02:02:30 INFO util.GSet: VM type= 64-bit
16/04/05 02:02:30 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
16/04/05 02:02:30 INFO util.GSet: capacity= 2^15 = 32768 entries
16/04/05 02:02:30 INFO namenode.FSImage: Allocated new BlockPoolId: BP-464058956-192.168.200.165-1459836150356
16/04/05 02:02:30 INFO common.Storage: Storage directory
/home/hadoop/hadoop/hdfs/name has been successfully formatted.
16/04/05 02:02:30 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/04/05 02:02:30 INFO util.ExitUtil: Exiting with status 0
16/04/05 02:02:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master1/192.168.200.165
************************************************************/
　　
8.启停hadoop
　　#启动hadoop集群
　　注意事项：
　　1）各个集群节点之间能免密码登录彼此
　　2）确保各节点配置的myid和zoo.cfg里的一致，并先启动zookeeper集群
　　启动命令：sbin/start-all.sh 或者依次执行 start-dfs.sh 和 start-yarn.sh



[hadoop@master1 conf]$ start-all.sh
Starting namenodes on [master1]
master1: starting namenode, logging to
/home/hadoop/hadoop/logs/hadoop-hadoop-namenode-master1.out
slave2: starting datanode, logging to
/home/hadoop/hadoop/logs/hadoop-hadoop-datanode-slave2.out
master2: starting datanode,
logging to /home/hadoop/hadoop/logs/hadoop-hadoop-datanode-master2.out
master1: starting datanode, logging to
/home/hadoop/hadoop/logs/hadoop-hadoop-datanode-master1.out
slave1: starting datanode, logging to
/home/hadoop/hadoop/logs/hadoop-hadoop-datanode-slave1.out
Starting secondary namenodes [master2]
master2: starting secondarynamenode, logging to
/home/hadoop/hadoop/logs/hadoop-hadoop-secondarynamenode-master2.out
starting yarn daemons
starting resourcemanager, logging to
/home/hadoop/hadoop/logs/yarn-hadoop-resourcemanager-master1.out
slave2: starting nodemanager, logging to
/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-slave2.out
master2: starting nodemanager, logging to
/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-master2.out
master1: starting nodemanager, logging to
/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-master1.out
slave1: starting nodemanager, logging to
/home/hadoop/hadoop/logs/yarn-hadoop-nodemanager-slave1.out
　　查看相关守护进程
　　NAMENODE（master1）



[hadoop@master1 logs]$ jps
12892 NameNode
13003 DataNode
13295 ResourceManager
13408 NodeManager
13826 QuorumPeerMain
　　SECONDARY NAMENODE（master2）



[hadoop@master2 ~]$ jps
10162 SecondaryNameNode
10052 DataNode
10245 NodeManager
4045 QuorumPeerMain
　　DATANODE（slave1/slave2）



[hadoop@slave1 ~]$ jps
13902 NodeManager
13789 DataNode
9331 QuorumPeerMain

[hadoop@slave2 ~]$ jps
13697 QuorumPeerMain
18324 DataNode
18440 NodeManager
　　#停止hadoop集群
　　sbin /stop-all.sh
　　启动成功后可以访问web控制台查看集群信息：
　　NAMENODE：http://192.168.200.165:50070/
　　SECONDARY NAMENODE：http://192.168.200.166:9001
　　Nodes Of Cluster（YARN作业管理界面）: http://192.168.200.166:8088
　　9.启动jobhistoryserver
　　因为kylin中需要MapReduce任务调度，所以需要启动jobhistoryserver



[hadoop@master1 logs]$ mr-jobhistory-daemon.sh start historyserver
　　查看jobhistoryserver守护进程



[hadoop@master1 conf]$ jps
24419 JobHistoryServer
　　
七、hbase 部署
　　1.解压缩hbase-1.1.4到hadoop家目录下，进入目录hbase
　　2.进入conf文件夹
　　3.修改hbase各配置文件如下：
　　hbase-env.sh



export JAVA_HOME=/jdk1.7.0_80
　　regionservers



master1
master2
slave1
slave2
　　hbase-site.xml（已优化）



<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master1:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master1,master2,slave1,slave2</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/zookeeper-3.4.6/data</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.master.info.bindAddress</name>
<value>master1</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>60010</value>
</property>
<property>
<name>hbase.master.maxclockskew</name>
<value>200000</value>
<description>Time difference of regionserver from master</description>
</property>
<property>
<name>hbase.coprocessor.user.region.classes</name>
<value>org.apache.hadoop.hbase.coprocessor.AggregateImplementation</value>
</property>
<property>
<name>hbase.regionserver.wal.codec</name>
<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property>
<property>
<name>hbase.master.loadbalancer.class</name>
<value>org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer</value>
</property>
<property>
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.phoenix.hbase.index.master.IndexMasterObserver</value>
</property>
<property>
<name>phoenix.query.maxServerCacheBytes</name>
<value>1073741824</value>
</property>
<property>
<name>phoenix.query.maxGlobalMemoryPercentage</name>
<value>70</value>
</property>
<property>
<name>hbase.client.scanner.caching</name>
<value>5000</value>
<description>HBase客户端扫描缓存，对查询性能有很大帮助</description>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>360000000</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>60000</value>
<description>zk超时时间</description>
</property>
<property>
<name>hbase.regionserver.handler.count</name>
<value>50</value>
<description>用户表接受外来请求的线程数</description>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>107374182400</value>
<description>单个ColumnFamily的region大小，若按照ConstantSizeRegionSplitPolicy策略，超过设置的该值则自动split>（100G）</description>
</property>
<property>
<name>perf.hfile.block.cache.size</name>
<value>0.2</value>
<description>设置读写平衡</description>
</property>
<property>
<name>hbase.regionserver.global.memstore.size</name>
<value>0.3</value>
<description>RegionServer进程block进行flush触发条件：该节点上所有region的memstore之和达到upperLimit*heapsize</description>
</property>
<property>
<name>hbase.regionserver.global.memstore.lowerLimit</name>
<value>0.3</value>
<description>RegionServer进程触发flush的一个条件：该节点上所有region的memstore之和达到lowerLimit*heapsize</description>
</property>
<property>
<name>hbase.zookeeper.property.tickTime</name>
<value>6000</value>
<description>Client端与zk发送心跳的时间间隔(6秒)</description>
</property>
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>10</value>
<description>设置读写平衡</description>
</property>
<property>
<name>hbase.hstore.blockingWaitTime</name>
<value>90000</value>
<description>block的等待时间（90s）</description>
</property>
<property>
<name>hbase.hregion.memstore.flush.size</name>
<value>104857600</value>
<description>memstore大小，当达到该值则会flush到外存设备（100M）</description>
</property>
<property>
<name>hbase.hregion.memstore.mslab.enabled</name>
<value>true</value>
<description>是否开启mslab方案，减少因内存碎片导致的Full GC，提高整体性能</description>
</property>
<property>
<name>hbase.regionserver.region.split.policy</name>
<value>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</value>
<description>split操作默认的策略</description>
</property>
<property>
<name>hbase.client.write.buffer</name>
<value>8388608</value>
<description>客户端写buffer，设置autoFlush为false时，当客户端写满buffer才flush（8m）</description>
</property>
<property>
<name>hbase.hregion.memstore.block.multiplier</name>
<value>4</value>
<description>如果memstores超过了flushsize的multiplier倍则会阻塞客户端的写</description>
</property>
<property>
<name>hbase.regionserver.regionSplitLimit</name>
<value>150</value>
<description>单台RegionServer上region数上限</description>
</property>
<property>
<name>hbase.regionserver.maxlogs</name>
<value>16</value>
<description>如果memstores超过了flushsize的multiplier倍则会阻塞客户端的写</description>
</property>
</configuration>
　　4.配置环境变量，优化内存



export HBASE_HOME=/home/hadoop/hbase
export PATH=$PATH:$HBASE_HOME/bin
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xms1536m -Xmx2048m -Xmn1024m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
-XX:PermSize=512m -XX:MaxPermSize=512m"

export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS
-Xms1536m -Xmx2048m -Xmn1024m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:PermSize=512m -XX:MaxPermSize=512m"

export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xms1024m -Xmx2048m"
　　5.分发hbase文件夹到各个虚拟主机上的hadoop家目录上



scp -r  ~/hbase/ hadoop@master2:~
scp -r  ~/hbase/ hadoop@slave1:~
scp -r  ~/hbase/ hadoop@slave2:~
　　6.启停



bin/start-hbase.sh
bin/stop-hbase.sh
　　启动成功信息



[hadoop@master1 hadoop]$ start-hbase.sh
master1: starting zookeeper, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-master1.out
slave2: starting zookeeper, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-slave2.out
master2: starting zookeeper, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-master2.out
slave1: starting zookeeper, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-zookeeper-slave1.out
starting master, logging to /home/hadoop/hbase/logs/hbase-hadoop-master-master1.out
slave2: starting regionserver, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-slave2.out
master1: starting regionserver, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-master1.out
master2: starting regionserver, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-master2.out
slave1: starting regionserver, logging to
/home/hadoop/hbase/bin/../logs/hbase-hadoop-regionserver-slave1.out
　　查看相关守护进程
　　HMASTER（master1）



[hadoop@master1 hadoop]$ jps
14230 HMaster
14379 HRegionServer
　　HREGIONSERVER（master2、slave1、slave2）



[hadoop@ master2 ~]$ jps
10574 HRegionServer
[hadoop@slave1 ~]$ jps
14230 HRegionServer
[hadoop@slave2 ~]$ jps
18753 HRegionServer
　　出现的问题：集群节点的时间不同步
　　org.apache.hadoop.hbase.ClockOutOfSyncException
　　设置时间同步
　　#yum install ntpdate
　　# ntpdate 0.asia.pool.ntp.org
　　#rm -rf /etc/localtime
#ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
　　查看时间
　　date
+%Y-%m-%d-%H:%M:%S

八、HIVE部署
　　仅在master1上部署即可


  • 解压缩hive-2.0.0到hadoop家目录下，进入目录hive


  • 进入conf文件夹


  • 在mysql上创建元数据库hive
　　在master1上创建数据库hive(编码选latin，如果不选择latin，会出现问题)

#为hive数据库授权




grant all on hive.* to 'root'@'%' IDENTIFIED BY 'weidong' with grant option;
flush privilege;
　　#设置mysql数据库为任意IP可连接



update user set host='%' where host='localhost';
　　4．修改hive各配置文件如下：
　　hive-site.xml



<configuration>
<property>
　　<name>javax.jdo.option.ConnectionURL</name>
　　　　<value>jdbc:mysql://master1:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>驱动名</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>用户名</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>weidong</value>
<description>密码</description>
</property>
<property>
<name>datanucleus.schema.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://master1:9000/home/hadoop/hive/warehouse</value>
<description>数据路径（相对hdfs）</description>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://master1:9000/home/hadoop/hive/warehouse</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/hive/logs</value>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/hadoop/hbase/lib</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://master1:9083</value>
<description>运行hive得主机地址及端口</description>
</property>
</configuration>
　　将日志配置打开



cp hive-log4j2.properties. template  hive-log4j2.properties
cp hive-exec-log4j2.properties.template  hive-exec-log4j2.properties
　　5．启动Hive
　　1）首先需要先启动元数据库



hive --service metastore &
　　启动成功信息



2016-04-06T11:46:10,157 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:main(5876)) - Starting hive metastore on port 9083
2016-04-06T11:46:10,210 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(499)) - 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2016-04-06T11:46:10,299 INFO  [main]:
metastore.ObjectStore (ObjectStore.java:initialize(318)) - ObjectStore, initialize called
2016-04-06T11:46:12,284 INFO  [main]:
metastore.ObjectStore (ObjectStore.java:getPMF(402)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2016-04-06T11:46:15,345 INFO  [main]:
metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(142)) - Using direct SQL, underlying DB is MYSQL
2016-04-06T11:46:15,352 INFO  [main]:
metastore.ObjectStore (ObjectStore.java:setConf(301)) - Initialized ObjectStore
2016-04-06T11:46:16,051 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(586)) - Added admin role in metastore
2016-04-06T11:46:16,058 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(595)) - Added public role in metastore
2016-04-06T11:46:16,239 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers_core(635)) - No user is added in admin role, since config is empty
2016-04-06T11:46:16,715 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6020)) - Starting DB backed MetaStore Server with SetUGI enabled
2016-04-06T11:46:16,729 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6077)) - Started the new metaserver on port [9083]...
2016-04-06T11:46:16,729 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6079)) - Options.minWorkerThreads = 200
2016-04-06T11:46:16,729 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6081)) - Options.maxWorkerThreads = 1000
2016-04-06T11:46:16,730 INFO  [main]:
metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6083)) - TCP keepalive = true
　　2）启动hive客户端
　　输入hive命令即可



[hadoop@master conf]$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hadoop/hive/lib/hive-jdbc-2.1.0-SNAPSHOT-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/hadoop/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/hadoop/spark/lib/spark-assembly-1.4.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Logging initialized using configuration in file:/home/hadoop/hive/conf/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
　　 检验hive客户端下能否查看/创建表
　　6．停止HIVE
　　ps –aux | grep hive 查看hive目前的进程PID，用kill杀掉即可。
　　7．常用命令



hive> show databases;
OK
default
Time taken: 1.881 seconds, Fetched: 1 row(s)
hive> use default;
OK
Time taken: 0.081 seconds
hive> create table kylin_test(test_count int);
OK
Time taken: 2.9 seconds
hive> show tables;
OK
kylin_test
Time taken: 0.151 seconds, Fetched: 1 row(s)
hive> select * from kylin_test;
OK
Time taken: 0.318 seconds
　　在hive数据库里查询

九、kylin部署
　　仅在master1上部署即可
　　1．了解kylin的两种二进制包
　　预打包的二进制安装包：apache-kylin-1.5.0-bin.tar.gz
　　特别二进制包：apache-kylin-1.5.0-HBase1.1.3-bin.tar.gz
　　
说明：特别二进制包是一个在HBase 1.1+环境上编译的Kylin快照二进制包；安装它需要HBase 1.1.3或更高版本，否则之前版本中有一个已知的关于fuzzy
key过滤器的缺陷，会导致Kylin查询结果缺少记录:HBASE-14269。此外还需注意的是，这不是一个正式的发布版(每隔几周rebase KYLIN 1.3.x 分支上最新的改动)，没有经过完整的测试。
　　2．解压缩apache-kylin-1.5.0-HBase1.1.3-bin.tar.gz到hadoop家目录下，进入目录kylin
　　3．在/etc/profile里配置KYLIN环境变量和一个名为hive_dependency的变量



export KYLIN_HOME=/home/hadoop/kylin
export PATH=$PATH:$ KYLIN_HOME/bin
export hive_dependency=/home/hadoop/hive/conf:/home/hadoop/hive/lib/*:/home/hadoop/hive/hcatalog/share/hcatalog/hive-hcatalog-core-2.0.0.jar
　　这个配置需要在从节点master2,slave1,slave2上同时配置，因为kylin提交的任务交给mr后，hadoop集群将任务分发给从节点时，需要hive的依赖信息，如果不配置，则mr任务将报错为: hcatalogXXX找不到。
　　4.修改kylin的启动脚本kylin.sh



1）显式声明 KYLIN_HOME
export KYLIN_HOME=/home/Hadoop/kylin
2）在HBASE_CLASSPATH_PREFIX中显示增加$hive_dependency依赖
export HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX
　　5．检查环境是否设置成功



[hadoop@master1 conf]$ check-env.sh
KYLIN_HOME is set to /home/hadoop/kylin
　　6．进入conf文件夹，修改kylin各配置文件如下：
　　kylin.properties



kylin.owner=wdcloud@kylin.apache.org
kylin.rest.servers=master1:7070
kylin.hdfs.working.dir=/home/hadoop/kylin/kylin_hdfs_working_dir
kylin.job.remote.cli.working.dir=/home/hadoop/kylin/kylin_job_working_dir
#定义kylin用于MR jobs的job.jar包和hbase的协处理jar包，用于提升性能。
kylin.job.jar=/home/hadoop/kylin/lib/kylin-job-1.5.0-SNAPSHOT.jar
kylin.coprocessor.local.jar=/home/hadoop/kylin/lib/kylin-coprocessor-1.5.0-SNAPSHOT.jar
　　
　　将kylin_hive_conf.xml和kylin_job_conf.xml的副本数设置为4



<property>
<name>dfs.replication</name>
<value>4</value>
<description>Block replication</description>
</property>
　　7．启动和停止kylin
　　#确认必须启动的服务：
　　1)hadoop2的hdfs/yarn/jobhistory服务
　　start-dfs.sh
　　start-yarn.sh
　　mr-jobhistory-daemon.sh start historyserver
2)hive 元数据库：hive --service metastore &

3)zookeeper

4)hbase ：start-hbase.sh

　　#检查hive和hbase的依赖



[hadoop@master1 kylin]$ find-hive-dependency.sh
[hadoop@master1 kylin]$ find-hbase-dependency.sh
　　#启动和停止kylin的命令如下：



[hadoop@master1 kylin]$ kylin.sh start
[hadoop@master1 kylin]$ kylin.sh stop
　　
Web访问地址
　　http://192.168.200.165:7070/kylin/login
　　默认的登录username/password 是 ADMIN/KYLIN.

十、kylin测试
　　1．测试Kylin自带的sample
　　Kylin提供一个自动化脚本来创建测试CUBE，这个脚本也会自动创建出相应的hive数据表。
　　运行sample例子的步骤：
　　① 运行${KYLIN_HOME}/bin/sample.sh脚本



[hadoop@master1 ~]$ sample.sh
　　关键提示信息：



KYLIN_HOME is set to /home/hadoop/kylin
Going to create sample tables in hive...
Sample hive tables are created successfully; Going to create sample cube...
Sample cube is created successfully in project 'learn_kylin'; Restart Kylin server or reload the metadata from web UI to see the change.
　　#在MYSQL中查看此sample创建了哪几张表
　　# select DB_ID,OWNER,SD_ID,TBL_NAME from TBLS;
　　#在hive客户端查看创建的表和数据量（1w条）



hive> show tables;
OK
kylin_cal_dt
kylin_category_groupings
kylin_sales
Time taken: 1.835 seconds, Fetched: 3 row(s)
hive> select count(*) from kylin_sales;
OK
10000
Time taken: 65.351 seconds, Fetched: 1 row(s)
　　② 重启kylin server 刷新缓存



[hadoop@master1 ~]$ kylin.sh stop
[hadoop@master1 ~]$ kylin.sh start
　　③ 使用默认的用户名密码ADMIN/KYLIN访问192.168.200.165:7070/kylin
　　进入控制台后选择project为learn_kylin的那个项目。
　　④ 选择测试cube “kylin_sales_cube”，点击“Action”-“Build”，选择一个2014-01-01以后的日期，这是为了选择全部的10000条测试记录。
　　选择一个生成日期
　　点击提交会出现重建任务成功提交的提示
　　⑤ 在监控台查看这个任务的执行进度，直到这个任务100%完成。
　　任务完成
　　切换到model控制台会发现cube的状态成为了ready，表示可以执行sql查询了
　　执行过程中，在hive里会生成临时表，待任务100%完成后，这张表会自动删除
　　kylin_intermediate_kylin_sales_cube_desc_20120201000000_20120201000000
　　执行过程中，在hbase里会生成永久的计算结果表，如：
　　KYLIN_PTQIXMC64A
　　如果build了两个以上的segment。还可以执行merge操作：
　　完成Merge任务
　　这时候HBASE里面不同的Segment表示的多张表也同时合并成了一张表，以节省磁盘空间
　　在Build过程中出现的问题：
　　当任务执行到第五步：创建HTable的时候，报错说创建的表不可用。
　　最终导致整个任务的失败 ERROR

　　2016-04-07 12:40:57,823 ERROR [pool-7-thread-5] steps.CubeHTableUtil:135 : Failed to create HTable
　　java.lang.IllegalArgumentException: table KYLIN_9USQAHQQXC created, but is not available due to some reasons
　　at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
　　at org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:132)
　　at org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:104)
　　at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
　　at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
　　at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
　　at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
　　at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
　　at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
　　at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
　　at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
　　at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
　　at java.lang.Thread.run(Thread.java:745)

　　出错的原因：
　　因为在kylin中默认使用了snappy压缩算法导致的。
　　HDFS报错日志：

　　2016-04-12 12:05:05,726 ERROR [RS_OPEN_REGION-slave2:16020-0]
　　handler.OpenRegionHandler: Failed open of
　　region=KYLIN_VKRC32OKFP,,1460433926913.73fb906719a75b2733f046e87fbe8105., starting to roll back the global memstore size.
　　org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test.
　　at org.apache.hadoop.hbase.util.CompressionTest.testCompression
　　(CompressionTest.java:91)
　　at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs
　　(HRegion.java:6300)
　　at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6251)
　　at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6218)
　　at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6189)
　　at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6145)
　　at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6096)
　　at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion
　　(OpenRegionHandler.java:362)
　　at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process
　　(OpenRegionHandler.java:129)
　　at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
　　at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
　　at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
　　at java.lang.Thread.run(Thread.java:745)
　　2016-04-12 12:05:05,727 INFO  [RS_OPEN_REGION-slave2:16020-0]
　　coordination.ZkOpenRegionCoordination: Opening of region {ENCODED =>
　　73fb906719a75b2733f046e87fbe8105, NAME =>
　　'KYLIN_VKRC32OKFP,,1460433926913.73fb906719a75b2733f046e87fbe8105.', STARTKEY => '', ENDKEY => '\x00\x01'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 1
　　2016-04-12 12:05:05,775 INFO  [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.RSRpcServices: Open
　　KYLIN_VKRC32OKFP,\x00\x01,1460433926913.06978b9fb1e423563a5aae7e1df044d8.

　　解决方法：禁用压缩或者使用LZO作为压缩算法
　　官网给出的禁用压缩算法的方法如下：
　　To disable compressing MR jobs you need to modify
　　$KYLIN_HOME/conf/kylin_job_conf.xml by removing all configuration entries related to compression(Just grep the keyword “compress”). To disable compressing hbase tables you need to open $KYLIN_HOME/conf/kylin.properties and remove the line starting with kylin.hbase.default.compression.codec.
　　⑥ 切换到Insight 窗口执行SQL语句，例如：

　　select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt;

　　在Kylin中执行如上的sql统计只用了0.46s (十次取平均值)
　　在Hive里执行同一条sql统计语句，花费时间高达136秒
　　hive> select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt;
　　Time taken: 136.489 seconds, Fetched: 731 row(s)
　　可见：kylin执行这条sql明显提升了效率。
　　其他测试语句：
　　①       select * from kylin_sales;(1s内)
　　②    各个时间段内的销售额及购买量（0.39秒）
　　select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers
　　from kylin_sales
　　group by part_dt
　　order by part_dt;
　　③    查询某一时间的销售额及购买量（0.40秒）
　　select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales
　　where part_dt = '2014-01-01'
　　group by part_dt;
　　发现报错：
　　Error while compiling generated Java code:
　　public static class Record3_0 implements java.io.Serializable {           
　　public java.math.BigDecimal f0;
　　public boolean f1;
　　public org.apache.kylin.common.hll.HyperLogLogPlusCounter f2;         
　　public Record3_0(java.math.BigDecimal f0, boolean f1, ...
　　这是因为part_dt是date类型，在解析string到date的时候出问题，应将sql语句改为：
　　select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers
　　from kylin_sales
　　where part_dt between '2014-01-01' and '2014-01-01'
　　group by part_dt;
　　或者
　　select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers
　　from kylin_sales
　　where part_dt = date '2014-01-01'
　　group by part_dt;
　　④上面查询只用到了fact table，而没有用到lookup table。如果查询各个时间段所有二级商品类型的销售额，则需要fact table与lookup table做inner join（1.36s）
　　select fact.part_dt, lookup.CATEG_LVL2_NAME, count(distinct seller_id) as sellers
　　from kylin_sales fact
　　inner join KYLIN_CATEGORY_GROUPINGS lookup
　　on fact.LEAF_CATEG_ID = lookup.LEAF_CATEG_ID and fact.LSTG_SITE_ID = lookup.SITE_ID
　　group by fact.part_dt, lookup.CATEG_LVL2_NAME
　　order by fact.part_dt desc

运维网声明 1、欢迎大家加入本站运维交流群:群②:261659950 群⑤:202807635 群⑦870801961 群⑧679858003
2、本站所有主题由该帖子作者发表，该帖子作者与运维网享有帖子相关版权
3、所有作品的著作权均归原作者享有，请您和我们一样尊重他人的著作权等合法权益。如果您对作品感到满意，请购买正版
4、禁止制作、复制、发布和传播具有反动、淫秽、色情、暴力、凶杀等内容的信息，一经发现立即删除。若您因此触犯法律，一切后果自负，我们对此不承担任何责任
5、所有资源均系网友上传或者通过网络收集，我们仅提供一个展示、介绍、观摩学习的平台，我们不对其内容的准确性、可靠性、正当性、安全性、合法性等负责，亦不承担任何法律责任
6、所有作品仅供您个人学习、研究或欣赏，不得用于商业或者其他用途，否则，一切后果均由您自己承担，我们对此不承担任何法律责任
7、如涉及侵犯版权等问题，请您及时通知我们，我们将立即采取措施予以解决
8、联系人Email：admin@iyunv.com 网址：www.yunweiku.com

所有资源均系网友上传或者通过网络收集,我们仅提供一个展示、介绍、观摩学习的平台,我们不对其承担任何法律责任,如涉及侵犯版权等问题,请您及时通知我们,我们将立即处理,联系人Email:kefu@iyunv.com,QQ:1061981298 本贴地址：https://www.yunweiku.com/thread-348681-1-1.html 上篇帖子： 深入分析 Java I/O 的工作机制 下篇帖子： Spark入门实战系列--8.Spark MLlib（下）--机器学习库SparkMLlib实战
回复

使用道具 举报

返回列表
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

扫码加入运维网微信交流群X

扫码加入运维网微信交流群

扫描二维码加入运维网微信交流群，最新一手资源尽在官方微信交流群！快快加入我们吧...

扫描微信二维码查看详情

客服 E-mail:iyunvcom@gmail.com

本站由青云提供云计算服务

运维网--中国最专业的运维工程师交流社区

豫ICP备20007574号-1 Copyright © 2012-2025

关于我们

发展历程

联系我们

关注我们

广告服务

广告案例

隐私条款

免责声明

用户登录

用户注册

版主守则

申请版主

手机版

金币获取

帮助中心

每日签到

Good Good

Study

Day Day

UP


客服E-mail:kefu@iyunv.com 客服QQ：1061981298


QQ群⑦：运维网交流群⑦ QQ群⑧：运维网交流群⑧ k8s群：运维网kubernetes交流群


提醒：禁止发布任何违反国家法律、法规的言论与图片等内容；本站内容均来自个人观点与网络等信息，非本站认同之观点.


本站大部分资源是网友从网上搜集分享而来，其版权均归原作者及其网站所有，我们尊重他人的合法权益，如有内容侵犯您的合法权益，请及时与我们联系进行核实删除！



合作伙伴: 青云cloud

快速回复 返回顶部 返回列表