Nutch开源搜索引擎与Paoding中文分词用plugin方式集成[转]

本文是我在集成中文分词paoding时积累的经验，单独成一篇文章来重点介绍，重点需要了解的有下面几个文件，a)插件目录及插件文件 build.xml,plugin.xml b)nutch-0.9\src\plugin\build.xml c)WEB-INF/classes/nutch-site.xml
然后通过按照下面的方式来配置，执行ant package就可以搞定了，这里用ant的方式来处理整个编译发布过程。

1)在src/plugin下面加入，analysis-zh和lib-paoding-analyzers目录。具体参见
E:\workspace\searchengine\nutch-0.9\src\plugin\analysis-zh
E:\workspace\searchengine\nutch-0.9\src\plugin\lib-paoding-analyzers

下面是analysis-zh中的源码，是对paoding的封装，代码超级easy,主要是把配置文件和ant脚本调对就可以了
/**
* Paoding chinese analyzer
*/

package org.apache.nutch.analysis.zh;

// JDK imp

orts
import java.io.Reader;

// Lucene imports
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;

// Nutch imports
import org.apache.nutch.analysis.NutchAnalyzer;


/**
* A simple Chinese Analyzer that wraps the Lucene one.
* @author kevin tu
*/
public class ChineseAnalyzer extends NutchAnalyzer {

private final static Analyzer ANALYZER =
new net.paoding.analysis.analyzer.PaodingAnalyzer();


/** Creates a new instance of ChineseAnalyzer */
public ChineseAnalyzer() { }


public TokenStream tokenStream(String fieldName, Reader reader) {
return ANALYZER.tokenStream(fieldName, reader);
}

}


2)修改src\plugin的build.xml
<target name="deploy">
<ant dir="analysis-zh" target="deploy"/><!--kevin 20080903 add-->
<ant dir="lib-paoding-analyzers" target="deploy"/><!--kevin 20080903 add-->
...
</target>

<target name="clean">
<ant dir="analysis-zh" target="clean"/><!--kevin 20080903 add-->
<ant dir="lib-paoding-analyzers" target="clean"/><!--kevin 20080903 add-->
...
</target>

3)修改nutch-site.xml,加入|analysis-(zh)| ，这个很重重要，否则nutch只会加载默认插件，不会加载paoding的jar包，和自己写的analysis-(zh) jar包
<property>
<name>plugin.includes</name>
<value>protocol-http|urlfilter-regex|parse-(text|html|js)|analysis-(zh)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
<description>
</description>
</property>

4)重新打包 ant package

5)配置tomcat,修改webapps/cse/WEB-INF/classes/nutch-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>local</value>
</property>

<property><!--指定本地的index目录-->
<name>searcher.dir</name>
<value>/nutch/local/crawled</value>
</property>
<property>
<name>plugin.includes</name>
<value>protocol-http|urlfilter-regex|parse-(text|html|js)|analysis-(zh)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
<description>
</description>
</property>
</configuration>

6)配置paoding运行参数,加入paoding-analysis.properties
paoding.imports=\
ifexists:classpath:paoding-analysis-default.properties;\
ifexists:classpath:paoding-analysis-user.properties;\
ifexists:classpath:paoding-knives-user.properties

配置export PAODING_DIC_HOME=/nutch/dic