【Mahout三】基于Mahout CBayes算法的20newsgroup流程分析

hujh20

1.Mahout环境搭建
　　1.下载Mahout

http://mirror.bit.edu.cn/apache/mahout/0.10.0/mahout-distribution-0.10.0.tar.gz
　　2.解压Mahout
　　3. 配置环境变量

vim /etc/profile
export HADOOP_HOME=/home/hadoop/software/hadoop-2.5.2
export PATH=$HADOOP_HOME/bin:$PATH
export MAHOUT_HOME=/home/hadoop/software/mahout-distribution-0.10.0
export PATH=$MAHOUT_HOME/bin:$PATH

　　4. Mahout依赖于Hadoop执行MapReduce作业，不同于一般的添加Hadoop依赖的方式，Mahout不需要修改它专有的配置文件进行Hadoop配置，只要在环境变量中添加HADOOP_HOME即可

2. Mahout Naive Bayes算法的执行流程
　　1. 针对样本文件创建Sequence File
2. 将Sequence File转换为tfidf向量文件
3. 根据tdidf向量文件创建training向量文件和测试向量文件
4. 训练Naive Bayes模型，输入为训练向量文件，输出为训练模型文件
5.1 基于训练向量文件进行测试
5.2 基于测试向量文件进行测试

3.具体流程

1.启动Mahout 20newsgroups程序
　　1. 运行脚本./examples/bin/classify-20newsgroups.sh

2.下载数据文件并上传到HDFS中
　　1. 数据文件（20news-bydate.tar.gz）下载到/tmp/mahout-work-hadoop目录下，并解压至本地/tmp/mahout-work-hadoop/20news-all目录下，目录下的数据文件是原始的文本内容.
　　2. 将数据文件上传到HDFS的/tmp/mahout-work-hadoop/20news-all目录下，执行如下HDFS命令

/home/hadoop/software/hadoop-2.5.2/bin/hdfs dfs -put /tmp/mahout-work-hadoop/20news-all /tmp/mahout-work-hadoop/
　　此时，HDFS上包含如下路径/tmp/mahout-work-hadoop/20news-all，这个目录下包含了20newsgroup的原始文件

3. 根据原始数据创建Sequence Files
　　1.通过执行如下的命令，为原始的数据20newsgroups创建Sequence Files

./bin/mahout seqdirectory -i /tmp/mahout-work-hadoop/20news-all -o /tmp/mahout-work-hadoop/20news-seq -ow
　　通过上面的命令可以看到，输入输出文件目录都是在HDFS上，产出的Sequence Files放于HDFS的/tmp/mahout-work-hadoop/20news-seq目录下，如下所示

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-seq
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:13 /tmp/mahout-work-hadoop/20news-seq/_SUCCESS
-rw-r--r--   1 hadoop supergroup   19202391 2015-05-22 07:13 /tmp/mahout-work-hadoop/20news-seq/part-m-00000
　　根据这个显示信息，mahout seqdirectory触发了Map任务，产生的part-m-00000文件的大小大约19M

4. 将Sequence Files转换为Vector
　　1.执行如下命令，将Sequence Files转换为向量，

./bin/mahout seq2sparse -i /tmp/mahout-work-hadoop/20news-seq -o /tmp/mahout-work-hadoop/20news-vectors -lnorm -nv -wt tfidf
　　通过上面的命令可知，生成的向量文件存放于HDFS的/tmp/mahout-work-hadoop/20news-vectors目录下
　　2.HDFS的/tmp/mahout-work-hadoop/20news-vectors目录下内容是

drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/df-count
-rw-r--r--   1 hadoop supergroup    1937084 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/dictionary.file-0
-rw-r--r--   1 hadoop supergroup    1890053 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/frequency.file-0
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:20 /tmp/mahout-work-hadoop/20news-vectors/tf-vectors
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:21 /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:14 /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents
drwxr-xr-x   - hadoop supergroup          0 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/wordcount
　　20news-vectors目录下有两个文件(dictionary.file-0和frequency.file.0)以及五个目录
　　2.1 df-count目录

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/df-count
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/df-count/_SUCCESS
-rw-r--r--   1 hadoop supergroup    1890073 2015-05-22 07:18 /tmp/mahout-work-hadoop/20news-vectors/df-count/part-r-00000
　　2.2 tf-vectors目录

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/tf-vectors
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:20 /tmp/mahout-work-hadoop/20news-vectors/tf-vectors/_SUCCESS
-rw-r--r--   1 hadoop supergroup   28689283 2015-05-22 07:20 /tmp/mahout-work-hadoop/20news-vectors/tf-vectors/part-r-00000
　　2.3 tfidf-vectors目录

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:21 /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors/_SUCCESS
-rw-r--r--   1 hadoop supergroup   28689283 2015-05-22 07:21 /tmp/mahout-work-hadoop/20news-vectors/tfidf-vectors/part-r-00000
　　2.4 tokenized-documents

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:14 /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents/_SUCCESS
-rw-r--r--   1 hadoop supergroup   27503580 2015-05-22 07:14 /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents/part-m-00000
　　2.5 wordcount

[hadoop@hadoop ~]$ hdfs dfs -ls /tmp/mahout-work-hadoop/20news-vectors/wordcount
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/wordcount/_SUCCESS
-rw-r--r--   1 hadoop supergroup    2315037 2015-05-22 07:15 /tmp/mahout-work-hadoop/20news-vectors/wordcount/part-r-00000
　　3. 以上2个文件和目录的生成过程
　　3.1 通过Map任务生成tokenized-documents

SparseVectorsFromSequenceFiles: Tokenizing documents in /tmp/mahout-work-hadoop/20news-seq
　　3.2 根据Tokenized-Documents生成Term Frequency Vectors
　　这是一个Map Reduce任务，生成的TFV保存在wordcount目录下

15/05/22 07:14:54 INFO SparseVectorsFromSequenceFiles: Creating Term Frequency Vectors
15/05/22 07:14:54 INFO DictionaryVectorizer: Creating dictionary from /tmp/mahout-work-hadoop/20news-vectors/tokenized-documents and saving at /tmp/mahout-work-hadoop/20news-vectors/wordcount
　　3.3 计算IDF
　　这是一个MapReduce任务，生成的文件放置于df-count目录下

15/05/22 07:17:34 INFO SparseVectorsFromSequenceFiles: Calculating IDF

3. 训练Bayes Model
　　通过如下命令生成Bayes Model

./bin/mahout trainnb -i /tmp/mahout-work-hadoop/20news-train-vectors -o /tmp/mahout-work-hadoop/model -li /tmp/mahout-work-hadoop/labelindex -ow -c
　　通过命令可以看出，为Bayes模型的训练提供的输入是/tmp/mahout-work-hadoop/20news-train-vectors,输出是/tmp/mahout-work-hadoop/model