ubuntu安装spark2.1 hadoop2.7.3集群

432edw

                      0: 设置系统登录相关
Master要执行

1	cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

如果用root用户

1	sed -ri 's/^(PermitRootLogin ).*$/\1yes/' /etc/ssh/sshd_config

编辑/etc/hosts

1 2 3 4 5 6 7 8 9 10 11

127.0.0.1       localhost   # 别把 spark1 放在这 192.168.100.25   spark1  #spark1 is Master 192.168.100.26   spark2 192.168.100.27   spark3 127.0.1.1       ubuntu # The following lines are desirable for IPv6 capable hosts ::1     localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters

如果把 spark1 放在/etc/hosts第一行, 会发现在slave 有下面的错误

1	org.apache.hadoop.ipc.Client: Retrying connect to server: spark1/192.168.100.25:9000. Already tried 0 time(s)

然后在spark1 运行

1 2	ss -lnt LISTEN      0      128             localhost:9000

会发现监听的是本地. 删除 hosts中的相关文本重新启动hadoop,解决问题


1: 安装java

可以直接apt-get

1 2 3 4

apt-get install python-software-properties -y add-apt-repository ppa:webupd8team/java apt-get update apt-get install oracle-java7-installer

或者下载

1 2 3 4 5 6 7 8 9 10 11 12 13

wget http://download.oracle.com/otn-p ... 80-linux-x64.tar.gz mkdir /usr/lib/jvm tar xvf jdk-7u80-linux-x64.tar.gz mv jdk1.7.0_80 /usr/lib/jvm # 配置相关路径 update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0_80/bin/java" 1 update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0_80/bin/javac" 1 update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0_80/bin/javaws" 1 update-alternatives --config java # 验证一下 java -version javac -version javaws -version

添加环境变量

1 2 3 4 5 6

cat >> /etc/profile <<EOF export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_80 export JRE_HOME=/usr/lib/jvm/jdk1.7.0_80/jre export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin EOF

2: 安装 hadoop

1 2 3 4

tar xvf hadoop-2.7.3.tar.gz mv hadoop-2.7.3 /usr/local/hadoop cd /usr/local/hadoop mkdir -p hdfs/{data,name,tmp}

添加环境变量

1 2 3 4

cat >> /etc/profile <<EOF export HADOOP_HOME=/usr/local/hadoop export PATH=$PATH:$HADOOP_HOME/bin EOF

编辑 hadoop-env.sh 文件

1	export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_80  #只改了这一行

编辑 core-site.xml 文件

1 2 3 4 5 6 7 8 9 10

<configuration>         <property>                 <name>fs.defaultFS</name>                 <value>hdfs://spark1:9000</value>         </property>         <property>                 <name>hadoop.tmp.dir</name>                 <value>/usr/local/hadoop/hdfs/tmp</value>         </property>         </configuration>

编辑 hdfs-site.xml 文件

1 2 3 4 5 6 7 8 9 10 11 12 13 14

<configuration>         <property>                 <name>dfs.namenode.name.dir</name>                 <value>/usr/local/hadoop/hdfs/name</value>         </property>         <property>                 <name>dfs.datanode.data.dir</name>                 <value>/usr/local/hadoop/hdfs/data</value>         </property>         <property>                 <name>dfs.replication</name>                 <value>3</value>         </property> </configuration>

编辑 mapred-site.xml 文件

1 2 3 4 5 6

<configuration>     <property>         <name>mapreduce.framework.name</name>         <value>yarn</value>     </property> </configuration>

编辑 yarn-site.xml 文件

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

<configuration>     <property>         <name>yarn.nodemanager.aux-services</name>         <value>mapreduce_shuffle</value>     </property>     <property>         <name>yarn.resourcemanager.hostname</name>         <value>spark1</value>     </property>     <!--property>         别添加这个属性,添加了可能出现下面的错误:         Problem binding to [spark1:0] java.net.BindException: Cannot assign requested address         <name>yarn.nodemanager.hostname</name>         <value>spark1</value>     </property--> </configuration>

上面相关文件的具体属性及值在官网查询:
https://hadoop.apache.org/docs/r2.7.3/

编辑 masters 文件

1	echo spark1 > masters

编辑 slaves 文件

1 2 3

spark1 spark2 spark3

安装好后,使用rsync 把相关目录及/etc/profile同步过去即可
启动hadoop dfs

1	./sbin/start-dfs.sh

初始化文件系统

1	hadoop namenode -format

启动 yarn

1	./sbin/start-yarn.sh

检查spark1相关进程

1 2 3 4 5 6 7

root@spark1:/usr/local/spark/conf# jps 1699 NameNode 8856 Jps 2023 SecondaryNameNode 2344 NodeManager 1828 DataNode 2212 ResourceManager

spark2 spark3 也要类似下面的运程

1 2 3 4

root@spark2:/tmp# jps 3238 Jps 1507 DataNode 1645 NodeManager

可以打开web页面查看

1	http://192.168.100.25:50070

测试hadoop

1 2 3 4

hadoop fs -mkdir /testin hadoop fs -put ~/str.txt /testin cd /usr/local/hadoop hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /testin/str.txt testout

结果如下:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /testin/str.txt testout 17/02/24 11:20:59 INFO client.RMProxy: Connecting to ResourceManager at spark1/192.168.100.25:8032 17/02/24 11:21:01 INFO input.FileInputFormat: Total input paths to process : 1 17/02/24 11:21:01 INFO mapreduce.JobSubmitter: number of splits:1 17/02/24 11:21:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1487839487040_0002 17/02/24 11:21:06 INFO impl.YarnClientImpl: Submitted application application_1487839487040_0002 17/02/24 11:21:06 INFO mapreduce.Job: The url to track the job: http://spark1:8088/proxy/application_1487839487040_0002/ 17/02/24 11:21:06 INFO mapreduce.Job: Running job: job_1487839487040_0002 17/02/24 11:21:28 INFO mapreduce.Job: Job job_1487839487040_0002 running in uber mode : false 17/02/24 11:21:28 INFO mapreduce.Job:  map 0% reduce 0% 17/02/24 11:22:00 INFO mapreduce.Job:  map 100% reduce 0% 17/02/24 11:22:15 INFO mapreduce.Job:  map 100% reduce 100% 17/02/24 11:22:17 INFO mapreduce.Job: Job job_1487839487040_0002 completed successfully 17/02/24 11:22:17 INFO mapreduce.Job: Counters: 49         File System Counters                 FILE: Number of bytes read=212115                 FILE: Number of bytes written=661449                 FILE: Number of read operations=0                 FILE: Number of large read operations=0                 FILE: Number of write operations=0                 HDFS: Number of bytes read=377966                 HDFS: Number of bytes written=154893                 HDFS: Number of read operations=6                 HDFS: Number of large read operations=0                 HDFS: Number of write operations=2         Job Counters                 Launched map tasks=1                 Launched reduce tasks=1                 Data-local map tasks=1                 Total time spent by all maps in occupied slots (ms)=23275                 Total time spent by all reduces in occupied slots (ms)=11670                 Total time spent by all map tasks (ms)=23275                 Total time spent by all reduce tasks (ms)=11670                 Total vcore-milliseconds taken by all map tasks=23275                 Total vcore-milliseconds taken by all reduce tasks=11670                 Total megabyte-milliseconds taken by all map tasks=23833600                 Total megabyte-milliseconds taken by all reduce tasks=11950080         Map-Reduce Framework                 Map input records=1635                 Map output records=63958                 Map output bytes=633105                 Map output materialized bytes=212115                 Input split bytes=98                 Combine input records=63958                 Combine output records=14478                 Reduce input groups=14478                 Reduce shuffle bytes=212115                 Reduce input records=14478                 Reduce output records=14478                 Spilled Records=28956                 Shuffled Maps =1                 Failed Shuffles=0                 Merged Map outputs=1                 GC time elapsed (ms)=429                 CPU time spent (ms)=10770                 Physical memory (bytes) snapshot=455565312                 Virtual memory (bytes) snapshot=1391718400                 Total committed heap usage (bytes)=277348352         Shuffle Errors                 BAD_ID=0                 CONNECTION=0                 IO_ERROR=0                 WRONG_LENGTH=0                 WRONG_MAP=0                 WRONG_REDUCE=0         File Input Format Counters                 Bytes Read=377868         File Output Format Counters                 Bytes Written=154893

3: 安装 scala

1 2	tar xvf scala-2.11.8.tgz mv scala-2.11.8 /usr/local/scala

添加环境变量

1 2 3 4

cat >> /etc/profile <<EOF export SCALA_HOME=/usr/local/scala export PATH=$PATH:$SCALA_HOME/bin EOF

测试

1 2 3

source /etc/profile scala -version Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL

4: 安装 spark

1 2	tar xvf spark-2.1.0-bin-hadoop2.7.tgz mv spark-2.1.0-bin-hadoop2.7 /usr/local/spark

添加环境变量

1 2 3 4 5

cat >> /etc/profile <<EOF export SPARK_HOME=/usr/local/spark export PATH=$PATH:$SPARK_HOME/bin export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native EOF

1 2 3

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native #这一条不添加的话在运行 spark-shell 时会出现下面的错误 NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

编辑 spark-env.sh

1 2	SPARK_MASTER_HOST=spark1 HADOOP_CONF_DIR=/usr/locad/hadoop/etc/hadoop

编辑 slaves

1 2 3

spark1 spark2 spark3

启动 spark

1	./sbin/start-all.sh

此时在spark1上运行jps应该如下, 多了 Master 和 Worker

1 2 3 4 5 6 7 8 9

root@spark1:/usr/local/spark/conf# jps 1699 NameNode 8856 Jps 7774 Master 2023 SecondaryNameNode 7871 Worker 2344 NodeManager 1828 DataNode 2212 ResourceManager

spark2 和 spark3 则多了 Worker

1 2 3 4 5

root@spark2:/tmp# jps 3238 Jps 1507 DataNode 1645 NodeManager 3123 Worker

可以打开web页面查看

1	http://192.168.100.25:8080/

运行 spark-shell

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

root@spark1:/usr/local/spark/conf# spark-shell Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/02/24 11:55:46 WARN SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0 17/02/24 11:56:17 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException Spark context Web UI available at http://192.168.100.25:4040 Spark context available as 'sc' (master = local , app id = local-1487908553475). Spark session available as 'spark'. Welcome to       ____              __      / __/__  ___ _____/ /__     _\ \/ _ \/ _ `/ __/  '_/    /___/ .__/\_,_/_/ /_/\_\   version 2.1.0       /_/           Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80) Type in expressions to have them evaluated. Type :help for more information. scala> :help

此时可以打开spark 查看

1	http://192.168.100.25:4040/environment/

至此完成.