spark 实战 1： 基于gettyimages spark docker image 创建spark 集群
1、首先下载该镜像到本地。https://hub.docker.com/r/gettyimages/spark/
~$ docker pull gettyimages/spark
2、从https://github.com/gettyimages/docker-spark/blob/master/docker-compose.yml下载编写好的支持 Spark 集群的 docker-compose.yml 文件
启动它
$ docker-compose up
$ docker-compose up
Creating spark_master_1
Creating spark_worker_1
Attaching to spark_master_1, spark_worker_1
master_1| 16/10/29 13:25:05 INFO master.Master: Started daemon with process name: 1@master
master_1| 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for TERM
master_1| 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for HUP
master_1| 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for INT
master_1| 16/10/29 13:25:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
master_1| 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls to: root
master_1| 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls to: root
master_1| 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls groups to:
master_1| 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls groups to:
master_1| 16/10/29 13:25:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; userswith view permissions: Set(root); groups with view permissions: Set(); userswith modify permissions: Set(root); groups with modify permissions: Set()
worker_1| 16/10/29 13:25:06 INFO worker.Worker: Started daemon with process name: 1@worker
worker_1| 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for TERM
worker_1| 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for HUP
worker_1| 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for INT
master_1| 16/10/29 13:25:07 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
master_1| 16/10/29 13:25:07 INFO master.Master: Starting Spark master at spark://master:7077
master_1| 16/10/29 13:25:07 INFO master.Master: Running Spark version 2.0.1
master_1| 16/10/29 13:25:07 INFO util.log: Logging initialized @2542ms
master_1| 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@509cf69e{/app,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@58b8a90e{/app/json,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3dbc763b{/,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54dbacbf{/json,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@634488bc{/static,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7e29bd03{/app/kill,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@207a1153{/driver/kill,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO server.ServerConnector: Started ServerConnector@569e522b{HTTP/1.1}{0.0.0.0:8080}
master_1| 16/10/29 13:25:07 INFO server.Server: Started @2742ms
master_1| 16/10/29 13:25:07 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
master_1| 16/10/29 13:25:07 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://localhost:8080
master_1| 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT
master_1| 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@40e10daa{/,null,AVAILABLE}
master_1| 16/10/29 13:25:07 INFO server.ServerConnector: Started ServerConnector@7d3ac291{HTTP/1.1}{master:6066}
master_1| 16/10/29 13:25:07 INFO server.Server: Started @2779ms
master_1| 16/10/29 13:25:07 INFO util.Utils: Successfully started service on port 6066.
master_1| 16/10/29 13:25:07 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
worker_1| 16/10/29 13:25:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
worker_1| 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls to: root
worker_1| 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls to: root
worker_1| 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls groups to:
worker_1| 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls groups to:
worker_1| 16/10/29 13:25:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; userswith view permissions: Set(root); groups with view permissions: Set(); userswith modify permissions: Set(root); groups with modify permissions: Set()
master_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@12d8a746{/metrics/master/json,null,AVAILABLE}
master_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@189c23e3{/metrics/applications/json,null,AVAILABLE}
master_1| 16/10/29 13:25:08 INFO master.Master: I have been elected leader! New state: ALIVE
worker_1| 16/10/29 13:25:08 INFO util.Utils: Successfully started service 'sparkWorker' on port 8881.
worker_1| 16/10/29 13:25:08 INFO worker.Worker: Starting Spark worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM
worker_1| 16/10/29 13:25:08 INFO worker.Worker: Running Spark version 2.0.1
worker_1| 16/10/29 13:25:08 INFO worker.Worker: Spark home: /usr/spark-2.0.1
worker_1| 16/10/29 13:25:08 INFO util.log: Logging initialized @2625ms
worker_1| 16/10/29 13:25:08 INFO server.Server: jetty-9.2.z-SNAPSHOT
worker_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1fcf4eb5{/logPage,null,AVAILABLE}
worker_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@10e1c732{/logPage/json,null,AVAILABLE}
worker_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6801b546{/,null,AVAILABLE}
worker_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@29814979{/json,null,AVAILABLE}
worker_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4c5e101a{/static,null,AVAILABLE}
worker_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@529896f4{/log,null,AVAILABLE}
worker_1| 16/10/29 13:25:08 INFO server.ServerConnector: Started ServerConnector@caf04fd{HTTP/1.1}{0.0.0.0:8081}
worker_1| 16/10/29 13:25:08 INFO server.Server: Started @2749ms
worker_1| 16/10/29 13:25:08 INFO util.Utils: Successfully started service 'WorkerUI' on port 8081.
worker_1| 16/10/29 13:25:08 INFO ui.WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://localhost:8081
worker_1| 16/10/29 13:25:08 INFO worker.Worker: Connecting to master master:7077...
worker_1| 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2366bb7{/metrics/json,null,AVAILABLE}
worker_1| 16/10/29 13:25:08 INFO client.TransportClientFactory: Successfully created connection to master/172.17.0.2:7077 after 41 ms (0 ms spent in bootstraps)
master_1| 16/10/29 13:25:09 INFO master.Master: Registering worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM
worker_1| 16/10/29 13:25:09 INFO worker.Worker: Successfully registered with master spark://master:7077
页:
[1]