Apache Tika源码研究（一）
因为采用Apache Tika解析网页文件时产生乱码问题，所以后来仔细看了一下Apache Tika源码先浏览一下tika编码识别的相关接口和类的UML模型
下面是编码识别接口，EncodingDetector.java
public interface EncodingDetector {
/**
* Detects the character encoding of the given text document, or
* null if the encoding of the document can not be detected.
*
* If the document input stream is not available, then the first
* argument may be null. Otherwise the detector may
* read bytes from the start of the stream to help in encoding detection.
* The given stream is guaranteed to support the
* {@link InputStream#markSupported() mark feature} and the detector
* is expected to {@link InputStream#mark(int) mark} the stream before
* reading any bytes from it, and to {@link InputStream#reset() reset}
* the stream before returning. The stream must not be closed by the
* detector.
*
* The given input metadata is only read, not modified, by the detector.
*
* @param input text document input stream, or null
* @param metadata input metadata for the document
* @return detected character encoding, or null
* @throws IOException if the document input stream could not be read
*/
Charset detect(InputStream input, Metadata metadata) throws IOException;
}
编码识别接口EncodingDetector的实现类有三，分别是HtmlEncodingDetector，UniversalEncodingDetector，和Icu4jEncodingDetector
从三者的名称基本可以看出他们的功能或所用的组件，Tika默认的网页编码识别是存在问题的，当解析网页文件时，在网页的html元素里面编码有误的时候就会产生乱码。
HtmlEncodingDetector类的源码如下：
public class HtmlEncodingDetector implements EncodingDetector {
// TIKA-357 - use bigger buffer for meta tag sniffing (was 4K)
private static final int META_TAG_BUFFER_SIZE = 8192;
private static final Pattern HTTP_EQUIV_PATTERN = Pattern.compile(
"(?is) 0 && !detector.isDone()) {
detector.handleData(buf, 0, nread);
}
detector.dataEnd();
}
如果我们的程序采用UniversalEncodingDetector类来识别文件编码，代码怎么实现呢？下面是调用方法：
public static void main(String[] args) throws IOException, TikaException {
// TODO Auto-generated method stub
File file=new File("[文件路径]");
InputStream stream=null;
try
{
stream=new FileInputStream(file);
EncodingDetectordetector=new UniversalEncodingDetector();
Charset charset = detector.detect(new BufferedInputStream(stream), new Metadata());
System.out.println("编码："+charset.name());
}
finally
{
if (stream != null) stream.close();
}
}
第三个类是Icu4jEncodingDetector，从名称可以看出是采用的IBM的Icu4j组件，代码如下：
public class Icu4jEncodingDetector implements EncodingDetector {
public Charset detect(InputStream input, Metadata metadata)
throws IOException {
if (input == null) {
return null;
}
CharsetDetector detector = new CharsetDetector();
String incomingCharset = metadata.get(Metadata.CONTENT_ENCODING);
String incomingType = metadata.get(Metadata.CONTENT_TYPE);
if (incomingCharset == null && incomingType != null) {
// TIKA-341: Use charset in content-type
MediaType mt = MediaType.parse(incomingType);
if (mt != null) {
incomingCharset = mt.getParameters().get("charset");
}
}
if (incomingCharset != null) {
detector.setDeclaredEncoding(CharsetUtils.clean(incomingCharset));
}
// TIKA-341 without enabling input filtering (stripping of tags)
// short HTML tests don't work well
detector.enableInputFilter(true);
detector.setText(input);
for (CharsetMatch match : detector.detectAll()) {
try {
return CharsetUtils.forName(match.getName());
} catch (Exception e) {
// ignore
}
}
return null;
}
}
最关键的类是CharsetDetector，这里暂不进一步分析这个类了
下面转帖网上的一篇博文
《使用ICU4J探测文档编码》
http://blog.iyunv.com/cnhome/article/details/6973343
曾经使用过这个东东，还是不错的，中国人的一篇论文，最早的时候好像是在哪个开源浏览器里。
网页源码的编码探测一般有两种方式，一种是通过分析网页源码中Meta信息，比如contentType，来取得编码，但是某些网页不的contentType中不含任何编码信息，这时需要通过第二种方式进行探测，第二种是使用统计学和启发式方法对网页源码进行编码探测。ICU4J就是基于第二种方式的类库。由IBM提供。
下面的例子演示了一个简单的探测过程。
package com.huilan.dig.contoller;
import java.io.IOException;
import java.io.InputStream;
import com.ibm.icu.text.CharsetDetector;
import com.ibm.icu.text.CharsetMatch;
/**
* 本类使用ICU4J包进行文档编码获取
*
*/
public class EncodeDetector {
/**
* 获取编码
* @throws IOException
* @throws Exception
*/
public static String getEncode(byte[] data,String url){
CharsetDetector detector = new CharsetDetector();
detector.setText(data);
CharsetMatch match = detector.detect();
String encoding = match.getName();
System.out.println("The Content in " + match.getName());
CharsetMatch[] matches = detector.detectAll();
System.out.println("All possibilities");
for (CharsetMatch m : matches) {
System.out.println("CharsetName:" + m.getName() + " Confidence:"
+ m.getConfidence());
}
return encoding;
}
public static String getEncode(InputStream data,String url) throws IOException{
CharsetDetector detector = new CharsetDetector();
detector.setText(data);
CharsetMatch match = detector.detect();
String encoding = match.getName();
System.out.println("The Content in " + match.getName());
CharsetMatch[] matches = detector.detectAll();
System.out.println("All possibilities");
for (CharsetMatch m : matches) {
System.out.println("CharsetName:" + m.getName() + " Confidence:"
+ m.getConfidence());
}
return encoding;
}
}
页:
[1]