[综合]Apache Hadoop 2.2.0集群安装(2)[翻译]

devil20

　　NodeManager节点健康监控
hadoop提供一个检测一个节点健康状态的机制，那就是管理员可以配置NodeManager去周期性执行一个脚本。
管理员可以在这个脚本中做任何的状态监控从而决定此节点是否健康。如果某节点不健康了，那么他们会有一个标准的错误输出，NodeManager的脚本周期性检测输出，如果节点输出中包含了ERROR字符串，那么此节点会被上报为unhealthy ，并且此节点会被ResourceManager放入黑名单。从而将不会有task被分配到此节点上，不过NodeManager 仍然会健康此节点，当此节点正常之后他将会被从ResourceManager 的黑名单中自动移除，节点的运行状况取决于输出，当他不正常的时候他仍然会在ResourceManager上展示。
如下参数为节点状况健康脚本的配置conf/yarn-site.xml：


Parameter
Value
Notes


yarn.nodemanager.health-checker.script.path
Node health script
Script to check for node's health status.


yarn.nodemanager.health-checker.script.opts
Node health script options
Options for script to check for node's health status.


yarn.nodemanager.health-checker.script.interval-ms
Node health script interval
Time interval for running health script.


yarn.nodemanager.health-checker.script.timeout-ms
Node health script timeout interval
Timeout for health script execution.




当一些物理磁盘出现坏道时监控程序不会提示错误。NodeManager 有能力对物理磁盘做周期性检测（特别是nodemanager-local-dirs and nodemanager-log-dirs）当目录损坏数达到配置的阀值(yarn.nodemanager.disk-health-checker.min-healthy-disks配置的)之后整个节点就会被标记为不正常的。同时这些信息也会上报给资源管理器(resource manager),检测脚本也会检测启动盘。
 
　　Slaves文件
通常你选择了一个机器做NameNode ，一个机器做ResourceManager，其他的做DataNode和NodeManager 也就是从节点。
把所有的从节点的ip或者hostname写在conf/slaves文件里，每个机器一行。
 
　　日志
Hadoop 用apache的log4j去访问Apache Commons Logging框架去记录日志。去修改conf/log4j.properties 可以自定义自己的日志输出。
 
　　操作Hadoop集群
一旦配置文件都已经配置完成之后拷贝他们到所有机器的HADOOP_CONF_DIR 目录
Hadoop启动
你需要启动hdfs和YARN 
格式化一个新的分布式系统：
 

$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>
　　在NameNode执行如下命令去启动hdfs：
 

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
　　在所有的从节点上执行如下命令启动DataNodes ：
 

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
　　在ResourceManager上执行如下命令去启动YARN

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
　　在所有的从节点上执行如下命令去启动NodeManagers ：
 

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
　　单独启动一个web服务器，如果需要负载均衡的话那么在每个机子上都执行如下脚本：
 

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver --config $HADOOP_CONF_DIR
　　在任何一台机子上执行如下命令去启动MapReduce JobHistory 服务：
 

$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
　　Hadoop集群关闭
在NameNode 节点上执行如下命令去关闭NameNode进程：
 

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode
　　在所有的从节点上执行如下脚本去停止DataNodes 进程：
 

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
　　在ResourceManager 节点上执行如下命令可以停止ResourceManager 进程：
 

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
　　在所有从节点执行如下命令去停止NodeManagers 进程：
 

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager
　　在运行WebAppProxy 的节点上执行如下命令可以停止WebAppProxy 服务：
 

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh stop proxyserver --config $HADOOP_CONF_DIR
　　在运行MapReduce JobHistory 服务的节点上执行如下命令去停止MapReduce JobHistory 服务：
 

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR
　　Hadoop在安全模式下运行
本节将讲述一些在安全模式下运行的参数，安全模式是可靠的基于Kerberos协议认证的。
Hadoop进程的用户账户
确保HDFS和YARN进程是由不同的Unix用户启动的，如hdfs，yarn，并且MapReduce JobHistory 是由mapred启动的。
推荐他们都属于同一个组如Hadoop：


User:Group
Daemons


hdfs:hadoop
NameNode, Secondary NameNode, Checkpoint Node, Backup Node, DataNode


yarn:hadoop
ResourceManager, NodeManager


mapred:hadoop
MapReduce JobHistory Server

HDSF和本地文件权限：
下表罗列出hdfs上的path和本地文件系统的推荐权限设置：


Filesystem
Path
User:Group
Permissions


local
dfs.namenode.name.dir
hdfs:hadoop
drwx------


local
dfs.datanode.data.dir
hdfs:hadoop
drwx------


local
$HADOOP_LOG_DIR
hdfs:hadoop
drwxrwxr-x


local
$YARN_LOG_DIR
yarn:hadoop
drwxrwxr-x


local
yarn.nodemanager.local-dirs
yarn:hadoop
drwxr-xr-x


local
yarn.nodemanager.log-dirs
yarn:hadoop
drwxr-xr-x


local
container-executor
root:hadoop
--Sr-s---


local
conf/container-executor.cfg
root:hadoop
r--------


hdfs
/
hdfs:hadoop
drwxr-xr-x


hdfs
/tmp
hdfs:hadoop
drwxrwxrwxt


hdfs
/user
hdfs:hadoop
drwxr-xr-x


hdfs
yarn.nodemanager.remote-app-log-dir
yarn:hadoop
drwxrwxrwxt


hdfs
mapreduce.jobhistory.intermediate-done-dir
mapred:hadoop
drwxrwxrwxt


hdfs
mapreduce.jobhistory.done-dir
mapred:hadoop
drwxr-x---

Kerberos Keytab文件：
HDFS：
NameNode 节点上的的keytab文件如下：
 

$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nn.service.keytab
Keytab name: FILE:/etc/security/keytab/nn.service.keytab
KVNO Timestamp         Principal
4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
　　Secondary NameNode 的keytab文件如下：
 

$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/sn.service.keytab
Keytab name: FILE:/etc/security/keytab/sn.service.keytab
KVNO Timestamp         Principal
4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
　　DataNode 的keytab文件如下：
 

$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/dn.service.keytab
Keytab name: FILE:/etc/security/keytab/dn.service.keytab
KVNO Timestamp         Principal
4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
　　YARN：
ResourceManager 节点上的ResourceManager keytab文件如下：
 

$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/rm.service.keytab
Keytab name: FILE:/etc/security/keytab/rm.service.keytab
KVNO Timestamp         Principal
4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
　　NodeManager节点上的keytab文件如下：
 

$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nm.service.keytab
Keytab name: FILE:/etc/security/keytab/nm.service.keytab
KVNO Timestamp         Principal
4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
　　MapReduce JobHistory Server：
MapReduce JobHistory Server keytab 文件如下：
 

$ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/jhs.service.keytab
Keytab name: FILE:/etc/security/keytab/jhs.service.keytab
KVNO Timestamp         Principal
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
　　

　　安全模式配置：
　　conf/core-site.xml：


Parameter
Value
Notes


hadoop.security.authentication
kerberos

simple is non-secure.


hadoop.security.authorization
true
Enable RPC service-level authorization.
　　conf/hdfs-site.xml：
　　NameNode配置：


Parameter
Value
Notes


dfs.block.access.token.enable
true
Enable HDFS block access tokens for secure operations.


dfs.https.enable
true
 


dfs.namenode.https-address
nn_host_fqdn:50470
 


dfs.https.port
50470
 


dfs.namenode.keytab.file
/etc/security/keytab/nn.service.keytab
Kerberos keytab file for the NameNode.


dfs.namenode.kerberos.principal
nn/_HOST@REALM.TLD
Kerberos principal name for the NameNode.


dfs.namenode.kerberos.https.principal
host/_HOST@REALM.TLD
HTTPS Kerberos principal name for the NameNode.
　　Secondary NameNode配置：


Parameter
Value
Notes


dfs.namenode.secondary.http-address
c_nn_host_fqdn:50090
 


dfs.namenode.secondary.https-port
50470
 


dfs.namenode.secondary.keytab.file
/etc/security/keytab/sn.service.keytab
Kerberos keytab file for the NameNode.


dfs.namenode.secondary.kerberos.principal
sn/_HOST@REALM.TLD
Kerberos principal name for the Secondary NameNode.


dfs.namenode.secondary.kerberos.https.principal
host/_HOST@REALM.TLD
HTTPS Kerberos principal name for the Secondary NameNode.
　　DataNode配置：


Parameter
Value
Notes


dfs.datanode.data.dir.perm
700
 


dfs.datanode.address
0.0.0.0:2003
 


dfs.datanode.https.address
0.0.0.0:2005
 


dfs.datanode.keytab.file
/etc/security/keytab/dn.service.keytab
Kerberos keytab file for the DataNode.


dfs.datanode.kerberos.principal
dn/_HOST@REALM.TLD
Kerberos principal name for the DataNode.


dfs.datanode.kerberos.https.principal
host/_HOST@REALM.TLD
HTTPS Kerberos principal name for the DataNode.
　　conf/yarn-site.xml：
　　WebAppProxy：
　　WebAppProxy在应用和用户之间提供了一个web输出，如果是在安全模式下那么当用户不安全访问的时候就会被警告，跟普通的web应用一样。


Parameter
Value
Notes


yarn.web-proxy.address

WebAppProxy host:port for proxy to AM web apps.

host:port if this is the same as yarn.resourcemanager.webapp.address or it is not defined then theResourceManager will run the proxy otherwise a standalone proxy server will need to be launched.


yarn.web-proxy.keytab
/etc/security/keytab/web-app.service.keytab
Kerberos keytab file for the WebAppProxy.


yarn.web-proxy.principal
wap/_HOST@REALM.TLD
Kerberos principal name for the WebAppProxy.
　　LinuxContainerExecutor：
　　YARN框架使用的ContainerExecutor 定义了多少个容器被启动和控制。
　　如下在Hadoop YARN是也是有效的：


ContainerExecutor
Description


DefaultContainerExecutor
The default executor which YARN uses to manage container execution. The container process has the same Unix user as the NodeManager.


LinuxContainerExecutor
Supported only on GNU/Linux, this executor runs the containers as the user who submitted the application. It requires all user accounts to be created on the cluster nodes where the containers are launched. It uses a setuid executable that is included in the Hadoop distribution. The NodeManager uses this executable to launch and kill containers. The setuid executable switches to the user who has submitted the application and launches or kills the containers. For maximum security, this executor sets up restricted permissions and user/group ownership of local files and directories used by the containers such as the shared objects, jars, intermediate files, log files etc. Particularly note that, because of this, except the application owner and NodeManager, no other user can access any of the local files/directories including those localized as part of the distributed cache.



　　构建LinuxContainerExecutor 执行如下脚本：

$ mvn package -Dcontainer-executor.conf.dir=/etc/hadoop/
　　通过 -Dcontainer-executor.conf.dir传过来的路径集群节点上必须有且是本地的路径，执行文件必须在$HADOOP_YARN_HOME/bin中有。执行文件必须有权限：6050 or --Sr-s--- ，NodeManager 的unix用户必须同组，这个组必须是个特殊的组，如果其他应用程序具有这个组的权限那么他将是不安全的，这个组的名称需要在 yarn.nodemanager.linux-container-executor.group 属性中配置涉及到conf/yarn-site.xml and conf/container-executor.cfg两个文件。
　　如：NodeManager 的启动用户为yarn 为hadoop组，users组中有如下两个用户yarn 和alice(应用程序提交者) 同时alice 不属于hadoop组如上所述那么setuid/setgid 执行文件必须设置权限为 6050 or --Sr-s--- ，yarn 用户和hadoop 组(这样alice 就不能执行了)。
　　LinuxTaskController 需要的目录 yarn.nodemanager.local-dirs andyarn.nodemanager.log-dirs他们的权限设置为755 。
　　conf/container-executor.cfg：
　　执行文件需要一个配置文件container-executor.cfg上面mvn提到的，此文件必须为运行NodeManager 的用户所有(如上面的yarn )，任意组那么权限为：0400 or r--------.
　　执行文件需要下属参数在conf/container-executor.cfg配置，以key-value对出现，并且一行一个。


Parameter
Value
Notes


yarn.nodemanager.linux-container-executor.group
hadoop
Unix group of the NodeManager. The group owner of the container-executor binary should be this group. Should be same as the value with which the NodeManager is configured. This configuration is required for validating the secure access of the container-executor binary.


banned.users
hfds,yarn,mapred,bin
Banned users.


allowed.system.users
foo,bar
Allowed system users.


min.user.id
1000
Prevent other super-users.



　　LinuxContainerExecutor中涉及到的本地文件系统权限如下：


Filesystem
Path
User:Group
Permissions


local
container-executor
root:hadoop
--Sr-s---


local
conf/container-executor.cfg
root:hadoop
r--------


local
yarn.nodemanager.local-dirs
yarn:hadoop
drwxr-xr-x


local
yarn.nodemanager.log-dirs
yarn:hadoop
drwxr-xr-x



　　 
　　ResourceManager配置：


Parameter
Value
Notes


yarn.resourcemanager.keytab
/etc/security/keytab/rm.service.keytab
Kerberos keytab file for the ResourceManager.


yarn.resourcemanager.principal
rm/_HOST@REALM.TLD
Kerberos principal name for the ResourceManager.
　　NodeManager配置：


Parameter
Value
Notes


yarn.nodemanager.keytab
/etc/security/keytab/nm.service.keytab
Kerberos keytab file for the NodeManager.


yarn.nodemanager.principal
nm/_HOST@REALM.TLD
Kerberos principal name for the NodeManager.


yarn.nodemanager.container-executor.class
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor
Use LinuxContainerExecutor.


yarn.nodemanager.linux-container-executor.group
hadoop
Unix group of the NodeManager.
　　conf/mapred-site.xml
　　MapReduce JobHistory Server配置：


Parameter
Value
Notes


mapreduce.jobhistory.address
MapReduce JobHistory Server host:port

Default port is 10020.


mapreduce.jobhistory.keytab
/etc/security/keytab/jhs.service.keytab
Kerberos keytab file for the MapReduce JobHistory Server.


mapreduce.jobhistory.principal
jhs/_HOST@REALM.TLD
Kerberos principal name for the MapReduce JobHistory Server.
　　操作hadoop集群
一旦配置完成之后就把所有HADOOP_CONF_DIR 里面的文件拷贝到其他节点上
此章节会说明不同的unix用户启动不同的hadoop服务，采用的unix系统用户和用户组
hadoop启动
启动hadoop集群你需要启动HDFS and YARN 集群
hdfs用户格式hadoop文件系统执行如下命令：

[hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>
在NameNode 节点上启动hdfs，用户为hdfs用户：

[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
在DataNodes 节点上启动DataNodes 用户为root，设置环境变量[size=1em]HADOOP_SECURE_DN_USER为hdfs：

[root]$ HADOOP_SECURE_DN_USER=hdfs $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
[size=1em]在ResourceManager 节点上执行如下命令启动YARN，用户为yarn：

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
在其他从节点上执行如下命令启动NodeManagers，用户为yarn：

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
用户yarn启动一个WebAppProxy 服务如果需要启动多个去负载均衡那么就用同样的方式启动多个：

[yarn]$ $HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR
用mapred用户启动MapReduce JobHistory Server ：

[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
 
hadoop集群关闭：
用户hdfs执行如下命令关闭NameNode ：

[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode
root用户在所有从节点上执行如下命令停止DataNodes ：

[root]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode
yarn用户在ResourceManager 节点上执行如下命令关闭ResourceManager：

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
yarn用户在所有的从节点上执行如下命令结束NodeManagers：

[yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager
yarn用户在WebAppProxy server.节点上执行如下命令停止WebAppProxy server.如果有多台那么依次：

[yarn]$ $HADOOP_YARN_HOME/bin/yarn stop proxyserver --config $HADOOP_CONF_DIR
mapred用户执行如下命令停止MapReduce JobHistory Server：

[mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR
 
　　Web监控页面
一旦集群启动之后可以通过web-ui监控进程运行情况：


Daemon
Web Interface
Notes


NameNode
http://nn_host:port/
Default HTTP port is 50070.


ResourceManager
http://rm_host:port/
Default HTTP port is 8088.


MapReduce JobHistory Server
http://jhs_host:port/
Default HTTP port is 19888.

          
[size=1em]