搭建Hadoop2.6+Hbase0.98.6+Nutch2.3环境
**编译Nutch并抓取网页
**
1、修改ivy/ivy.xml
<dependency org="org.apache.gora" name="gora-core" rev="0.6" conf="*->default"/>
<!---->//取消该注释
<dependency org="org.apache.gora" name="gora-hbase" rev="0.6" conf="*->default" />
<dependency org="org.apache.gora" name="gora-compiler-cli" rev="0.6" conf="*->default"/>
<dependency org="org.apache.gora" name="gora-compiler" rev="0.6" conf="*->default"/>
将hadoop1.2相关的去掉，然后添加：
<dependency org="org.apache.hadoop" name="hadoop-client" rev="2.6.0" conf="*->default"/>
2、修改ivysetting.xml
编译时部分jar包不能下载，需要修改如下配置。
<property name="repository.apache.org" value="http://maven.restlet.org/" override="false"/>
3、修改nutch-site.xml
<configuration>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.hbase.store.HBaseStore</value>
<description>Default class for storing data</description>
</property>
<property>
<name>http.agent.name</name>
<value>My Nutch Spider</value>
</property>
<property>
<name>io.serializations</name>
<value>org.apache.hadoop.io.serializer.WritableSerialization</value>
<description>A list of serialization classes that can be used for
obtaining serializers and deserializers.</description>
</property>
</configuration>
4、修改gora.properties
gora.datastore.default=org.apache.gora.hbase.store.HBaseStore
5、编译
ant runtime
6、添加缺失jar包
拷贝commons-compress-1.9.jar，vorbis-java-core-0.6.jar，nekohtml-1.9.13.jar到/runtime/local/lib/下
7、创建seek文件
在/runtime/local/下创建urls文件夹，并创建seek.txt，内容为：
http://nutch.apache.org/
8、上传urls文件夹到HDFS
hadoop fs -copyFromLocal urls .
9、抓取网页
bin/crawl seed crawl 5
10、查看HBase数据库
scan 'crawl_webpage'
常见问题
1、**org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)问题
nutch-site.xml新增配置**
<property>
<name>io.serializations</name>
<value>org.apache.hadoop.io.serializer.WritableSerialization</value>
<description>A list of serialization classes that can be used for
obtaining serializers and deserializers.</description>
</property>
版权声明：本文为博主原创文章，未经博主允许不得转载。
页:
[1]