spark过程

231211

scala配置
1、下载解压包
tar -xvf scala-2.10.4.tgz -C /usr/local/
2、包重命名为scala
3、配置环境变量

export SCALA_HOME=/usr/local/scala
export PATH=$PATH:/usr/local/scala/bin
4、执行生效source /etc/profile
##验证配置
scala -version 得到
Scala code runner version 2.10.4 -- Copyright 2002-2013, LAMP/EPFL

如果得到以上这句话，恭喜你，scala配置成功！


maven配置

1、下载解压包
tar -xvf apache-maven-3.3.9-bin.tar.gz -C /usr/local/
2、包重命名为maven
3、配置环境变量/etc/profile
export MAVEN_HOME=/usr/local/maven
export PATH=$PATH:/usr/local/maven/bin
export MAVEN_OPTS="-Xms256m -Xmx512m"
##验证配置
mvn -v 得到
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /usr/local/maven
Java version: 1.7.0_55, vendor: Oracle Corporation
Java home: /usr/local/jdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.el6.x86_64", arch: "i386", family: "unix"

如果得到以上这句话，恭喜你，scala配置成功！

安装编译spark
1、解压源码包：tar -zxvf spark-2.0.2-bin-hadoop2.7.tgz -C /usr/local/
cd /usr/local/
mv spark-2.0.2-bin-hadoop2.7 spark-2.0.2
source /etc/profile
2、复制配置模板文件
cd /usr/local/spark-2.0.2/conf
cp spark-env.sh.template spark-env.sh
cp slaves.template slaves
cp spark-defaults.conf.template spark-defaults.conf
主要配置JAVA_HOME、SCALA_HOME、HADOOP_HOME、HADOOP_CONF_DIR、SPARK_MASTER_IP等
vim spark-env.sh
export JAVA_HOME=/usr/local/jdk
export SCALA_HOME=/usr/local/scala
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_LAUNCH_WITH_SCALA=0
export SPARK_WORKER_MEMORY=1g
export SPARK_DRIVER_MEMORY=1g
export SPARK_MASTER_IP=192.168.1.114
export SPARK_LIBRARY_PATH=/usr/local/spark-2.0.2/lib
export SPARK_MASTER_WEBUI_PORT=18080
export SPARK_WORKER_DIR=/home/spark
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_PORT=7078
export SPARK_LOG_DIR=/home/spark_log
export SPARK_PID_DIR='/home/spark/run'

slaves（将所有节点都加入，master节点同时也是worker节点）

spark-defaults.conf
spark.master                     yarn-client
spark.home                       /root/spark-without-hive
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://Goblin01:8020/spark-log
spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.executor.memory            1g
spark.driver.memory              1g
spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.master指定Spark运行模式，可以是yarn-client、yarn-cluster...

spark.home指定SPARK_HOME路径

spark.eventLog.enabled需要设为true

spark.eventLog.dir指定路径，放在master节点的hdfs中，端口要跟hdfs设置的端口一致（默认为8020），否则会报错

spark.executor.memory和spark.driver.memory指定executor和dirver的内存，512m或1g，既不能太大也不能太小，因为太小运行不了，太大又会影响其他服务

配置yar-site.xml,跟hdfs-site.xml在同一个路径下（$HADOOP_HOME/etc/hadoop)
ll /usr/local/hadoop/etc/hadoop/yarn-site.xml

</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>haproxy:8030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>haproxy:8035</value>
</property>
<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>mycat:8033</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>mycat:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>


</configuration>
把spark-2.0.2复制到其他节点
启动start-all.sh


7. 运行

1) 准备一个文本文件放在/logs/wordcount.log内容为:



2) 运行spark-shell