Hive ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.RegexSerDe

jokerchu

在Hive的测试中，发现使用正则表达式的字段分隔方式时，总是进行不下去，创建表的语句如下：
create table mylog2(time string,clause string) row format serde 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' with serdeproperties (
"input.regex" = "(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2},\\d{3}) - (.*) -(.*)"
);

　　通过查看Tracking URL，发现抛出类找不到的异常，类

org.apache.hadoop.hive.contrib.serde2.RegexSerDe没有找到。　　

　　解决方法：
　　将下面的配置语句，加在配置文件： $HIVE_INSTALL/conf/hive-site.xml中，value中hive-contrib-*.jar的路径为你机器上实际的放置，在$HIVE_INSTALL/lib目录下寻找。

　　<property>

<name>hive.aux.jars.path</name>

<value>file:///home/develop/hive-0.8.0/lib/hive-contrib-0.8.0.jar</value>

<description>Added by tiger.zeng on 20120202.These JAR file are available to all users for all jobs</description>

</property>

　　退出hive shell, 重新启动hive shell, 可运行正常。

运行结果如下：　　hive> select time,clause from mylog2;

Total MapReduce jobs = 1

Launching Job 1 out of 1

Number of reduce tasks is set to 0 since there's no reduce operator

Starting Job = job_201202021642_0003, Tracking URL = http://hdp0:50030/jobdetails.jsp?jobid=job_201202021642_0003

Kill Command = /home/hadoop/hadoop-0.20.203.0/bin/hadoop job -Dmapred.job.tracker=hdp0:9001 -kill job_201202021642_0003

Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0

2012-02-02 16:58:43,327 Stage-1 map = 0%, reduce = 0%

2012-02-02 16:58:49,375 Stage-1 map = 100%, reduce = 0%

2012-02-02 16:58:55,437 Stage-1 map = 100%, reduce = 100%

Ended Job = job_201202021642_0003

MapReduce Jobs Launched:

Job 0: Map: 1 HDFS Read: 1286 HDFS Write: 564 SUCESS

Total MapReduce CPU Time Spent: 0 msec

OK

NULL NULL

NULL NULL

NULL NULL

NULL NULL

2012-01-17 00:14:19,301 org.apache.hadoop.conf.Configuration -0 [main] DEBUG org.apache.hadoop.conf.Configuration

NULL NULL

NULL NULL

NULL NULL

NULL NULL

NULL NULL

2012-01-17 00:14:19,314 org.apache.hadoop.util.NativeCodeLoader -13 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader

2012-01-17 00:14:19,314 org.apache.hadoop.util.NativeCodeLoader -13 [main] INFO org.apache.hadoop.util.NativeCodeLoader

2012-01-17 00:14:19,316 org.apache.hadoop.io.compress.zlib.ZlibFactory -15 [main] INFO org.apache.hadoop.io.compress.zlib.ZlibFactory

NULL NULL

Time taken: 21.28 seconds

hive>



　　参考：
　　

Jasper Knulst 13020438802011年04月06日 星期三 06时51分20秒

Link Here Print view

Hi(ve),
I created a table like this;
create table testtable (veld1 STRING,veld2 STRING,veld3 STRING) ROW FORMAT
SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
> WITH SERDEPROPERTIES ("input.regex" =
"([a-z]{4}[0-9])þ([a-z]{4}[0-9])þ([a-z]{4}[0-9])") STORED AS TEXTFILE;

The table is OK, select * from testtable shows the contents of the
underlying HDFS file.
However when I invoke a MR job by select veld2 from testtable, the MR job
starts but I get mapper errors saying:
"Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassNotFoundException:
org.apache.hadoop.hive.contrib.serde2.RegexSerDe"
I already copied the hive serde jar (in my case hive-serde-0.7.0-CDH3B4.jar)
to $HADOOP_HOME/lib and restarted jobtracker/tasktrackers but that doesn't
help.
Cheers JasperShow footer--
Kind Regards \ Met Vriendelijke Groet,


Jasper Knulst
BI Consultant


VLC Den Haag
Gildeweg 5B
2632 BD  Nootdorp

M: +31 (0)6 19 66 75 11
T: +31 (0)15 764 07 50

Loren Siebert 13020443192011年04月06日 星期三 06时58分39秒

Link Here Print view

You need to tell Hive about the JAR. This is how I do it in hive-site.xml:
<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-contrib-0.7.0-CDH3B4.jar</value>
<description>These JAR file are available to all users for all jobs</description>
</property>Show quoted text