大数据系列3：用Python编写MapReduce

yunde110

      vi mapper.py
      输入：
             #!/usr/bin/env python
 
             importsys
 
             for linein sys.stdin:
                           line= line.strip()
                           words= line.split()
                           forword in words:
                                         print'%s\t%s' % (word,1)
      chmod +x mapper.py
      vi reducer.py
      输入：
            #!/usr/bin/envpython
 
            from operator import itemgetter
 
            import sys
 
            current_word = None
            current_count = 0
            word = None
 
            for line in sys.stdin:
                          line = line.strip()
                          word, count =line.split('\t', 1)
                          try:
                                        count =int(count)
                          except ValueError:
                                        continue
                          if current_word ==word:
                                        current_count+= count
                          else:
                                        ifcurrent_word:
                                                      print'%s\t%s' % (current_word, current_count)
                                        current_count= count
                                        current_word= word
            if current_word:
                           print'%s\t%s' % (current_word, current_count)
      chmod +x reducer.py
      本地操作系统测试：
      echo "foo fooquux labs foo bar quux" | ./mapper.py | sort | ./reducer.py
      提交HADOOP集群运行：
      hadoop jar hadoop-1.2.1/contrib/streaming/hadoop-streaming-1.2.1.jar -input input -output output-streaming-python  -mapper /home/ysc/mapper.py -reducer /home/ysc/reducer.py
 
 
 

 
APDPlat旗下十大开源项目