Solr: integrate carrot2 with solr-5.1.0

lchl0388

　　I already integrated carrot2 with solr-4.x with my customerized chinese tokenizer successfully.
　　But I run some errors following my series of blogs http://ylzhj02.iyunv.com/blog/2152348  to adopt carrot2 to solr-5.1.0
　　The error is

org.carrot2.util.factory.FallbackFactory; Tokenizer for Chinese Simplified (zh_cn) is not available. This may degrade clustering quality of Chinese Simplified content. Cause: java.lang.NoSuchMethodError: org.apache.lucene.analysis.Tokenizer.<init>(Ljava/io/Reader;)V

　　The reason is that solr-5.2.1 adopted lucene 5.1.0, however carrot2-3.10.0  used lucene 4.6.0.   So the cause is jars uncompatible.

So, the solution is to download the latest version of carrot2

#git clone git://github.com/carrot2/carrot2.git

(3.11.0)

the lucene version is now 5.1.0

#cd carrot2

 

step 1:

#vi core/carrot2-util-text/src/org/carrot2/text/linguistic/DefaultTokenizerFactory.java

add

import org.carrot2.text.linguistic.lucene.InokChineseTokenizerAdapter;

 

change

100         map.put(LanguageCode.CHINESE_SIMPLIFIED,
101             new NewClassInstanceFactory<ITokenizer>(ChineseTokenizerAdapter.class));

 to

map.put(LanguageCode.CHINESE_SIMPLIFIED,
new NewClassInstanceFactory<ITokenizer>(InokChineseTokenizerAdapter.class));
 

step 2:

#vi InokChineseTokenizerAdapter.java

#cp chineseTokenizer/InokChineseTokenizerAdapter.java ./core/carrot2-util-text/src/org/carrot2/text/linguistic/lucene/

step 3:

#mkdir lib/org.lionsoul.jcseg

├── build.properties
├── jcseg-core-1.9.6.jar
├── jcseg.LICENSE
└── META-INF
    └── MANIFEST.MF

the file and jars is

build.properties

bin.includes = META-INF/,\
jcseg-core-1.9.6.jar,\
jcseg.LICENSE
META-INF/MANIFEST.MF

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Jcseg Tokenizer
Bundle-SymbolicName: org.lionsoul.jcseg
Bundle-Version: 1.9.6
Bundle-ClassPath: jcseg-core-1.9.6.jar
Bundle-Vendor: INokNok Inc.
Bundle-RequiredExecutionEnvironment: JavaSE-1.6
 

step 4:

modify build.xml

141   <patternset id="lib.test">
142     <include name="core/**/*.jar" />
143     <include name="lib/**/*.jar" />
144     <include name="lib/org.lionsoul.jcseg/*.jar" />
145     <exclude name="lib/org.slf4j/slf4j-nop*" />
146     <include name="applications/carrot2-dcs/**/*.jar" />
147     <include name="applications/carrot2-webapp/lib/*.jar" />
148     <include name="applications/carrot2-benchmarks/lib/*.jar" />
149   </patternset>

 
173   <patternset id="lib.core">
174     <include name="lib/**/*.jar" />
175     <include name="lib/org.lionsoul.jcseg/*.jar" />
176     <include name="core/carrot2-util-matrix/lib/*.jar" />
177     <patternset refid="lib.core.excludes" />
178   </patternset>

 

180   <patternset id="lib.core.mini">
181     <include name="lib/**/mahout-*.jar" />
182     <include name="lib/**/jcseg*.jar" />
183     <include name="lib/**/mahout.LICENSE" />
184     <include name="lib/**/colt.LICENSE" />
185     <include name="lib/**/commons-lang*" />
186     <include name="lib/**/guava*" />
187     <include name="lib/**/jackson*" />
188     <include name="lib/**/lucene-snowball*" />
189     <include name="lib/**/lucene.LICENSE" />
190     <include name="lib/**/hppc-*.jar" />
191     <include name="lib/**/hppc*.LICENSE" />
192     
193     <include name="lib/**/slf4j-api*.jar" />
194     <include name="lib/**/slf4j-nop*.jar" />
195     <include name="lib/**/slf4j.LICENSE" />
196     
197     <include name="lib/**/attributes-binder-*.jar" />
198   </patternset>
199   

 

906   <target name="core" depends="jar, jar.src, lib-no-jar.flattened" description="Builds Carrot2 Java API JAR with dependencies">
907     <delete dir="${api.dir}" failonerror="false" />
908     <mkdir dir="${api.dir}" />
909     <mkdir dir="${api.dir}/lib" />
910     <mkdir dir="${api.dir}/examples" />
911     <mkdir dir="${api.dir}/resources" />
912
913     <patternset id="carrot2.required">
914       <include name="**/jcseg*" />
915       <include name="**/commons-lang*" />

 

step 6:

#ant jar

#scp tmp/jar/carrot2-core-3.11.0-SNAPSHOT.jar root@192.168.0.135:/opt/solr/contrib/clustering/lib
carrot2-core-3.11.0-SNAPSHOT.jar

restart solr server to test clustering

 -----------------------------

An error happans

org.apache.solr.common.SolrException; null:java.lang.RuntimeException: java.lang.NoClassDefFoundError: com
/carrotsearch/hppc/ObjectHashSet

 

Solution :

#scp lib/com.carrotsearch.hppc/hppc-0.7.1.jar root@192.168.0.135:/opt/solr/contrib/clustering/lib/
hppc-0.7.1.jar                                       

#rm -f  opt/solr/contrib/clustering/lib/hppc-0.5.2.jar                                       

------

another error is

java.lang.RuntimeException: java.lang.IllegalAccessError: class
com.carrotsearch.hppc.ObjectHashSet cannot access its superclass com.carrotsearch.hppc.AbstractObjectCollection
 

The reason is that there is an old hppc-0.5.2.jar in /opt/solr/server/webapps/solr.war 


so, Solution is to

#cd /opt/solr/server/solr-webapp/webapp

#rm -f WEB-INF/lib/hppc-0.5.2.jar

#cp hppc-0.7.1.jar   WEB-INF/lib


#jar cf solr.war  ./

#mv solr.war  /opt/solr/server/webapps

restart solr

the error disappers