DCE需要NodeManager节点上运行着docker服务和docker client用于启动docker container。为了避免超时,作业依赖的docker镜像必须在NodeManager启动前预先下载好,运行以下命令下载hadoop docker镜像:
sudo docker pull sequenceiq/hadoop-docker:2.4.1同时在yarn-site.xml中增加配置:
<property>
<name>yarn.nodemanager.docker-container-executor.exec-name</name>
<value>/usr/bin/docker</value>
<description>
Name or path to the Docker client. This is a required parameter. If this is empty,
user must pass an image name as part of the job invocation(see below).
</description>
</property>
<property>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor</value>
<description>
This is the container executor setting that ensures that all
jobs are started with the DockerContainerExecutor.
</description>
</property>需要集群管理员注意目前DCE不提供用户命名空间隔离,这意味着在Yarn container中以root用户运行的程序会获得NodeManager的root权限,换句话说,DCE目前没有提供比Yarn默认的Container Executor更好的安全保证。
登录公共docker
repository,执行:
docker login [OPTIONS] [SERVER]
Register or log in to a Docker registry server, if no server is specified
"https://index.docker.io/v1/" is the default.
-e, --email="" Email
-p, --password="" Password
-u, --username="" Username
mapreduce.map.env: You can override the mapper’s image by passing yarn.nodemanager.docker-container-executor.image-name=your_image_name to this JVM property.
mapreduce.reduce.env: You can override the reducer’s image by passing yarn.nodemanager.docker-container-executor.image-name=your_image_name to this JVM property.
yarn.app.mapreduce.am.env: You can override the ApplicationMaster’s image by passing yarn.nodemanager.docker-container-executor.image-name=your_image_name to this JVM property.
五、docker镜像要求 Yarn
container使用的docker镜像有以下要求:
Docker镜像系统的发行版和版本可以和Nodemanager不同,然而,如果你在使用MapReduce计算框架,那么你的镜像需要配置可以运行hadoop,必须安装java以及定义以下环境变量JAVA_HOME,
HADOOP_COMMON_PATH, HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME, HADOOP_YARN_HOME, and HADOOP_CONF_DIR。