hadoop shell

tedwhy

查看集群状态：
hadoop dfsadmin -report
　　显示块的信息，包括：备份数量，存在哪个节点，保存状态
hadoop fsck /user/ak47/tmp_0.gz -files -racks -blocks
　　检查HDFS状态：
hadoop fsck {path}

举例：
hadoop fsck /user
　　使用distcp在集群间进行文件复制（在dist机器执行）
hadoop distcp hdfs://{src_host_ip}:9000/log/src/FCACCESS/20110105/00 hdfs://{dist_hostname}:9000/log/src/FCACCESS/20110105/00
　　手动 关闭/打开 服务
hadoop-daemon.sh start/stop jobtracker/namenode/datanode
　　关闭安全模式：
hadoop dfsadmin -safemode leave
　　马上开始一个测试Mapreduce job
hadoop-0.20 fs -mkdir input
hadoop-0.20 fs -put /etc/hadoop-0.20/conf/*.xml input
hadoop-0.20 fs -ls input
hadoop-0.20 jar /usr/lib/hadoop-0.20/hadoop-*-examples.jar grep input output 'dfs[a-z.]+'


冒烟测试
测试HDFS性能
hadoop jar hadoop-test*.jar TestDFSIO -write -nrFiles 10 -fileSize 1000 # 测试write
hadoop jar hadoop-test*.jar TestDFSIO -read -nrFiles 100 -fileSize 100 # 测试read
hadoop jar hadoop-test*.jar TestDFSIO -clean # 清除生成数据
排序测试
hadoop jar hadoop-examples-*.jar teragen 10000000 /user/hadoop/input_dir
hadoop jar hadoop-examples-*.jar terasort /user/hadoop/input_dir /user/hadoop/output_dir

#4台内存32GB、8core，1T*6磁盘、map slot 32个、reduce slot 16个，耗时37s
求PI值
hadoop jar hadoop-examples-*.jar pi 10 100