Spring for Apache Hadoop

352262

　　简介：Spring for Apache Hadoop provides  integration with the Spring Framework to create and run Hadoop MapReduce, Hive,  and Pig jobs as well as work with HDFS and HBase. If you have simple needs to  work with Hadoop, including basic scheduling, you can add the Spring for Apache  Hadoop namespace to your Spring based project and get going quickly using  Hadoop. As the complexity of your Hadoop application increases, you may want to  use Spring Batch and Spring Integration to regin in the complexity of developing  a large Hadoop application.
　　1.Using  the Spring for Apache Hadoop Namespace
　　To use the SHDP namespace, one just needs to import  it inside the configuration:
　　

           　　

　　

　　Spring for Apache Hadoop namespace prefix. Any name  can do but through out the reference documentation, the
　　
hdp
　　
will be  used.

　　The namespace URI.

　　The namespace URI location. Note that even though  the location points to an external address (which exists and is valid), Spring  will resolve the schema locally as it is included in the Spring for Apache  Hadoop library.

　　Declaration example for the Hadoop namespace. Notice  the prefix usage.
　　
2. Configuring Hadoop
　　In order to use Hadoop, one needs to first configure it namely by creating  a Configuration object. The configuration holds information about the job  tracker, the input, output format and the various other parameters of the map  reduce job.
　　

　　

　　

使用xml配置文件：　　

　　

　　

　　

使用properties配置文件：　　

                             fs.default.name=hdfs://localhost:9000          hadoop.tmp.dir=/tmp/hadoop          electric=sea         　　

　　

使用表达式避免硬编码（properties文件）：　　

　　

                             fs.default.name=${hd.fs}          hadoop.tmp.dir=file://${java.io.tmpdir}          hangar=${number:18}                            　　

　　
3.Creating a Hadoop Job
　　example：
　　未指定Configuration，会默认使用hadoopConfiguration
　　

　　

　　

Notice that there is no reference to the Hadoop configuration above - that's because, if not specified, the default naming convention (hadoopConfiguration) will be used instead.　　

指定Configuration配置文件：　　

　　

　　

4.Creating a Hadoop Streaming Job　　

　　

　　

　　

5.Running a Hadoop Job　　

for basic job submission SHDP provides the job-runner element (backed    by JobRunner>　　

　　

　　

　　

Multiple jobs can be specified and even nested if they are not used outside the runner:　　

　　

Do note that the runner will not run unless triggered manually or if run-at-startup is set to true　　

　　

6.Using the Hadoop Job tasklet　　

　　

　　

wait-for-job is true so that the tasklet will wait for the job to complete when it executes　　

7.Running a Hadoop Tool　　

hadoop jar带参运行：　　

　　
bin/hadoop jar -conf hadoop-site.xml -jt darwin:50020 -D property=value someJar.jar org.foo.SomeTool data/in.txt data/out.txt
　　

　　

　　
spring for hadoop 使用tool-runner运行带参的mapreduce