Param's Blog: Nutch 1.3 and Solr Integration

candy

Param's Blog: Nutch 1.3 and Solr Integration

Nutch 1.3 and Solr Integration
This post lists out steps to integrate Apache Nutch 1.3 and Solr.
Prerequisites
To Setup Nutch, please check Apache Nutch 1.3 Setup for details.
To Setup Solr, please check Apache Solr 1.4 Setup for details.
Integration
Step 1: Copy NUTCH_HOME/runtime/local/conf/schema.xml to SOLR_HOME/example/solr/conf directory.
Step 2: Restart the Solr server for new schema to take effect.
Step 3: Run below mentioned command to index already crawled data into Solr.
bin/nutch solrindex http://localhost:8080/solr crawl/crawldb crawl/linkdb crawl/segments/*
Tip: Check the crawl directory name which is created inside runtime/local folder. In my case, the name was crawl-20110719111306

Or you can crawl and index with single command:

bin/nutch crawl urls -solr http://localhost:8080/solr -threads 3 -depth 3 -topN 10
Issues Faced:
1. Getting the below mentioned error while indexing on Solr, the fix is to make sure that you are using correct solrj client library version in both nutch and Solr server. Adding the correct version runtime/local/lib will solve the poroblem:
java.io.IOException: Job failed!
SolrDeleteDuplicates: starting at 2011-07-19 10:41:08
SolrDeleteDuplicates: Solr url: http://localhost:8983/solr
Exception in thread "main" java.io.IOException: org.apache.solr.client.solrj.SolrServerException: Error executing query... 8 more
Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 1) or the data in not in 'javabin' format
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
... 10 more
Other Installation related Posts
Metrics Plugin for Eclipse
Installing PMD plugin in Eclipse
Maven Dependency in Eclipse
Setting up URL rewrite module on Apache
Setting up Apache and Tomcat for web application deployment
Installing and setting up Apache on Linux Machine
Basic Development Environment Setup on Mac
MongoDB Installation
Hadoop Setup Tutorial
Apache Nutch Setup
Apache Solr Setup
Posted by param at 8:37 AM
Labels: apache, installation, nutch, solr
8 comments:
Anonymous said...
    How did you update solrj when you say "Adding the correct version runtime/local/lib will solve the poroblem:"? Didi you just copy the jar files from a solr installation?
    July 24, 2011 3:50 PM
param said...
    Yup. Just copy the solrj jar file from solr installation (it will be inside dist folder.)
    July 28, 2011 2:35 PM
christopher said...
    Hi, thanks for this post. I am still getting the error after copying apache-solr-solrj-1.4.1 out of solr/dist into /local/lib
    SolrDeleteDuplicates: Solr url: http://localhost:8080/solr
    Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
    at org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDuplicates.java:362)
    at org.apache.nutch.crawl.Crawl.run(Crawl.java:152)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.nutch.crawl.Crawl.main(Crawl.java:54)
    I have tried it by deleting the original nutch solrj-1.3 and also by renaming apache-solr-solrj without the apache prefix.
    Is there a config file that I should set for it to find the new solrj file?
    August 1, 2011 8:53 AM
param said...
    hey chris, just ensuring you need to copy solrj lib inside /runtime/local/lib directory. Also, remove the original 1.3 solrj from this location. It should resolve the problem. I did not update the name of the file "apache-solr-solrj-1.4.1.jar", it automatically referenced the correct jar.
    August 1, 2011 11:18 AM
Godfrey Obinchu said...
    Or make sure that solr is running when crawling url using Nutch
    August 3, 2011 6:11 PM
param said...
    Or the url given in crawl command is not proper for example
    bin/nutch crawl urls -solr http://localhost:8983/ -depth 4 -topN 5 -threads 4
    instead of
    bin/nutch crawl urls -solr http://localhost:8983/solr -depth 4 -topN 5 -threads 4
    August 4, 2011 12:13 PM
Anonymous said...
    Thank you, your tip about proper version of solrj jar saved me lots of time!
    /Michael
    September 28, 2011 2:34 PM
param said...
    Hey Michael, I am glad it helped :).
    September 29, 2011 7:35 AM