[hadoop@bigdata ~]$ hadoop fs -copyFromLocal -f start-hadoop.sh hdfs://bigdata.lzf:9001/log
[-blockpools <comma-separated list of blockpool>
[-idleiterations <idleiterations>]
详细参考
http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
运行集群均衡
非常重要命令
由于各种原因,需要重新均衡数据节点。例如添加了新节点之后
cacheadmin hdfs cacheadmin -addDirective -path <path> -pool <pool-name> [-force] [-replication <replication>] [-ttl <time-to-live>]
详细参考
http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html#cacheadmin_command-line_interface
缓存管理
非常重要命令
官方微了这个写了一大篇的文章进行描述。
datanode hdfs datanode [-regular | -rollback | -rollingupgrade rollback] 数据节点管理
用于启动数据节点和滚动升级中进行回滚
dfsadmin hdfs dfsadmin [GENERIC_OPTIONS]
[-report [-live] [-dead] [-decommissioning]]
[-safemode enter | leave | get | wait | forceExit]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true |false |check]
[-refreshNodes]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> [-storageType <storagetype>] <dirname>...<dirname>]
[-clrSpaceQuota [-storageType <storagetype>] <dirname>...<dirname>]
[-finalizeUpgrade]
[-rollingUpgrade [<query> |<prepare> |<finalize>]]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh <host:ipc_port> <key> [arg1..argn]]
[-reconfig <datanode |...> <host:ipc_port> <start |status>]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-getBalancerBandwidth <datanode_host:ipc_port>]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-fetchImage <local directory>]
[-shutdownDatanode <datanode_host:ipc_port> [upgrade]]
[-getDatanodeInfo <datanode_host:ipc_port>]
[-evictWriters <datanode_host:ipc_port>]
[-triggerBlockReport [-incremental] <datanode_host:ipc_port>]
[-help [cmd]]
文件管理
核心命令--至关重要
haadmin hdfs haadmin -checkHealth <serviceId> hdfs haadmin -failover [--forcefence] [--forceactive] <serviceId> <serviceId>
hdfs haadmin -getServiceState <serviceId>
hdfs haadmin -help <command>
hdfs haadmin -transitionToActive <serviceId> [--forceactive]
hdfs haadmin -transitionToStandby <serviceId>
高可靠管理
核心命令-至关重要
journalnode hdfs journalnode
参考
http://blog.csdn.net/kiwi_kid/article/details/53514314
http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Administrative_commands
http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html
http://blog.csdn.net/dr_guo/article/details/50975851 --搭建ha集群参考
运行一个名称节点见同步服务mover hdfs mover [-p <files/dirs> | -f <local file name>]
参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html#Mover_-_A_New_Data_Migration_Tool
运行数据迁移。用于迁移压缩文件。类似于均衡器。定时均衡有关数据
namenode hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReserved<k-v pairs>] ] |
[-upgradeOnly [-clusterid cid] [-renameReserved<k-v pairs>] ] |
[-rollback] |
[-rollingUpgrade <rollback |started> ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby [-force] [-nonInteractive] [-skipSharedEditsCheck] ] |
[-recover [-force] ] |
[-metadataVersion ]
名称节点管理(
核心命令-至关重要)
进行备份,格式化,升级,回滚,恢复等等至关重要的操作。
nfs3 hdfs nfs3
参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html#Start_and_stop_NFS_gateway_service
启动一个nfs3网关,能够以类似操作系统文件浏览方式来浏览hdfs文件。
通过这个东西,有的时候能够更方便地操作
portmap hdfs portmap
参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html#Start_and_stop_NFS_gateway_service
和nfs服务器一起使用secondarynamenode hdfs secondarynamenode [-checkpoint [force]] | [-format] | [-geteditsize]
参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Secondary_NameNode
关于第二名称节点
storagepolicies hdfs storagepolicies
参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html
压缩存储策略管理
在某些环境下很有利。也许以后不存在所谓ssd的问题,仅仅是内存还是磁盘的问题
zkfc hdfs zkfc [-formatZK [-force] [-nonInteractive]]
参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Administrative_commands
管理动物园管理员节点
和journalnoe有关
高可靠的重要部分
debug hdfs debug verifyMeta -meta <metadata-file> [-block <block-file>] 检验hdfs的元数据和块文件。 hdfs debug computeMeta -block <block-file> -out <output-metadata-file>
谨慎使用,官方告警:
Use at your own risk! If the block file is corrupt and you overwrite it’s meta file, it will show up as ‘good’ in HDFS,
but you can’t read the data. Only use as a last measure, and when you are 100% certain the block file is good.
通过块文件计算元数据
hdfs debug recoverLease -path <path> [-retries <num-retries>] 恢复租约?
恢复特定路径的租约
第三部分 yarn命令
细节参考 http://hadoop.apache.org/docs/r2.8.0/hadoop-yarn/hadoop-yarn-site/YarnCommands.html
下表列出命令概览
yarn命令概览命令 语法和概述备注 applicationyarn application [options]打开应用报告或者终止应用 applicationattemptyarn applicationattempt [options]打印应用尝试报告 classpathyarn>yarn daemonlog -getlevel <host:httpport> <classname>