spark 实战 1： 基于gettyimages spark docker image 创建spark 集群

yanfangsheng123

　　

　　1、首先下载该镜像到本地。https://hub.docker.com/r/gettyimages/spark/
　　~$ docker pull gettyimages/spark
　　

　　2、从https://github.com/gettyimages/docker-spark/blob/master/docker-compose.yml下载编写好的支持 Spark 集群的 docker-compose.yml 文件
　　

　　启动它
　　$ docker-compose up
　　$ docker-compose up
　　Creating spark_master_1
　　Creating spark_worker_1
　　Attaching to spark_master_1, spark_worker_1
　　master_1  | 16/10/29 13:25:05 INFO master.Master: Started daemon with process name: 1@master
　　master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for TERM
　　master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for HUP
　　master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for INT
　　master_1  | 16/10/29 13:25:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
　　master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls to: root
　　master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls to: root
　　master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls groups to:
　　master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls groups to:
　　master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
　　worker_1  | 16/10/29 13:25:06 INFO worker.Worker: Started daemon with process name: 1@worker
　　worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for TERM
　　worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for HUP
　　worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for INT
　　master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
　　master_1  | 16/10/29 13:25:07 INFO master.Master: Starting Spark master at spark://master:7077
　　master_1  | 16/10/29 13:25:07 INFO master.Master: Running Spark version 2.0.1
　　master_1  | 16/10/29 13:25:07 INFO util.log: Logging initialized @2542ms
　　master_1  | 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@509cf69e{/app,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@58b8a90e{/app/json,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3dbc763b{/,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54dbacbf{/json,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@634488bc{/static,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7e29bd03{/app/kill,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@207a1153{/driver/kill,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO server.ServerConnector: Started ServerConnector@569e522b{HTTP/1.1}{0.0.0.0:8080}
　　master_1  | 16/10/29 13:25:07 INFO server.Server: Started @2742ms
　　master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
　　master_1  | 16/10/29 13:25:07 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://localhost:8080
　　master_1  | 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT
　　master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@40e10daa{/,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:07 INFO server.ServerConnector: Started ServerConnector@7d3ac291{HTTP/1.1}{master:6066}
　　master_1  | 16/10/29 13:25:07 INFO server.Server: Started @2779ms
　　master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service on port 6066.
　　master_1  | 16/10/29 13:25:07 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
　　worker_1  | 16/10/29 13:25:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
　　worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls to: root
　　worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls to: root
　　worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls groups to:
　　worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls groups to:
　　worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
　　master_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@12d8a746{/metrics/master/json,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@189c23e3{/metrics/applications/json,null,AVAILABLE}
　　master_1  | 16/10/29 13:25:08 INFO master.Master: I have been elected leader! New state: ALIVE
　　worker_1  | 16/10/29 13:25:08 INFO util.Utils: Successfully started service 'sparkWorker' on port 8881.
　　worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Starting Spark worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM
　　worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Running Spark version 2.0.1
　　worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Spark home: /usr/spark-2.0.1
　　worker_1  | 16/10/29 13:25:08 INFO util.log: Logging initialized @2625ms
　　worker_1  | 16/10/29 13:25:08 INFO server.Server: jetty-9.2.z-SNAPSHOT
　　worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1fcf4eb5{/logPage,null,AVAILABLE}
　　worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@10e1c732{/logPage/json,null,AVAILABLE}
　　worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6801b546{/,null,AVAILABLE}
　　worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@29814979{/json,null,AVAILABLE}
　　worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4c5e101a{/static,null,AVAILABLE}
　　worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@529896f4{/log,null,AVAILABLE}
　　worker_1  | 16/10/29 13:25:08 INFO server.ServerConnector: Started ServerConnector@caf04fd{HTTP/1.1}{0.0.0.0:8081}
　　worker_1  | 16/10/29 13:25:08 INFO server.Server: Started @2749ms
　　worker_1  | 16/10/29 13:25:08 INFO util.Utils: Successfully started service 'WorkerUI' on port 8081.
　　worker_1  | 16/10/29 13:25:08 INFO ui.WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://localhost:8081
　　worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Connecting to master master:7077...
　　worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2366bb7{/metrics/json,null,AVAILABLE}
　　worker_1  | 16/10/29 13:25:08 INFO client.TransportClientFactory: Successfully created connection to master/172.17.0.2:7077 after 41 ms (0 ms spent in bootstraps)
　　master_1  | 16/10/29 13:25:09 INFO master.Master: Registering worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM
　　worker_1  | 16/10/29 13:25:09 INFO worker.Worker: Successfully registered with master spark://master:7077