hadoop2.2+hive0.13安装和配置

奥飞火千万电

　　很久没用过hive了，前几天把hadoop的测试环境全部铲了，需要重新部署，于是下了个hive 0.13. 把过程记录一下：
　　1. 下载后解压tar包到/hive目录。
　　2. 进到/hive/conf下面，vi一个hive-site.xml文件，配置内容如下：
　　<configuration>

        <property>

                <name>javax.jdo.option.ConnectionURL</name>

                <value>jdbc:mysql://h1:3306/hive?createDatabaseIfNotExist=true</value>

        </property>

        <property>

                <name>javax.jdo.option.ConnectionDriverName</name>

                <value>com.mysql.jdbc.Driver</value>

        </property>

        <property>

                <name>javax.jdo.option.ConnectionUserName</name>

                <value>hadoop</value>

        </property>

        <property>

                <name>javax.jdo.option.ConnectionPassword</name>

                <value>123</value>

        </property>

        <property>

                <name>hive.metastore.local</name>

                <value>true</value>

        </property>

        <property>

                <name>hive.metastore.warehouse.dir</name>

                <value>hdfs://h1:9000/hive/warehouse</value>

        </property>

</configuration>  


　　3. 将mysql的连接器mysql-connector-java-5.1.30-bin.jar拷贝到/hive/lib目录下。


　　4. 在hdfs上创建/hive/warehouse目录。
　　5. 此时尝试执行了下/hive/bin/hive命令，提示异常:Access denied for user 'hadoop'@'h1', 这是由于mysql里面没有给hadoop用户，赋权。在mysql中执行：
　　grant all privileges on hive.* to hadoop@h1 identified by &quot;123&quot;, 再次执行/hive/bin/hive, 异常消失。
　　6. 此时在执行hive命令时，会出现以下提示：
　　[hadoop@h1 /hive/conf]$/hive/bin/hive

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

14/05/02 12:11:37 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed

14/05/02 12:11:37 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.


　　上述提示是由HIVE的bug引起的，详情见https://issues.apache.org/jira/browse/HIVE-6159，另外实际上在该0.13中该bug已经修复，但由于官网下载的hive版本是基于hadoop 0.20平台编译的，当前hadoop 2.2环境下需要重新编译hive. 在没重新编译之前，执行show databases;的命令，提示报错如下：
　　hive> show databases;

OK

Failed with exception java.io.IOException:java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!

Time taken: 0.444 seconds
　　7. 打开hive的调试模式，运行：hive -hiveconfhive.root.logger=DEBUG,console
　　可以看到报出的异常如下：
　　14/05/03 11:24:52 ERROR CliDriver: Failed with exception java.io.IOException:java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!

java.io.IOException: java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!

        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:551)

        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:489)

        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)

        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1471)

        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)

        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)

        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)

        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)

        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)

        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork!

        at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:223)

        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:379)

        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:515)

        ... 14 more

Caused by: java.lang.RuntimeException: Error in configuring object

        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

        at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:219)

        ... 16 more

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)

        ... 19 more

Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)

        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)

        at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)

        ... 24 more

Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found

        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)

        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)

        ... 26 more
　　可以看到该报错是由于没有找到lzo的类引起的。
　　8. 将hadoop-lzo-0.4.15.jar拷贝到/hadoop/lib下面，在hadoop-env.sh中将/hadoop/lib/hadoop-lzo-0.4.15.jar添加到hadoop_classpath中。
　　再次运行/hive/bin/hive, 不再提示报错。但提示以下：
　　[hadoop@h1 ~]$/hive/bin/hive

14/05/03 10:53:04 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

14/05/03 10:53:04 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

14/05/03 10:53:04 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

14/05/03 10:53:04 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

14/05/03 10:53:04 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

14/05/03 10:53:04 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

14/05/03 10:53:04 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative



Logging initialized using configuration in jar:file:/hive/lib/hive-common-0.12.0.jar!/hive-log4j.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/hive/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

hive>


　　


　　9. 除了以上，为了在hadoop 2.0平台上重新编译hive, 下载源码后，利用mvn编译，执行命令：
　　

#mvn clean install -Phadoop-2,dist
提示报错如下：
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure:
[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1688,30] getHadoopConfNames() is already defined in org.apache.hadoop.hive.conf.HiveConf
[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1707,30] getHadoopConfNames() is already defined in org.apache.hadoop.hive.conf.HiveConf
[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1669,3] method does not override or implement a method from a supertype
[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1687,3] method does not override or implement a method from a supertype
[ERROR] /hive/hive-src/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1706,3] method does not override or implement a method from a supertype
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure
        at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:858)
        at org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:129)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)

查看几个关键类HiveConf.java，ShimLoader, HadoopShims的源码，实在没找到问题出现在哪里，有没有高手指点一二，以上报错什么原因？
　　
　　


　　


　　


　　



版权声明：本文为博主原创文章，未经博主允许不得转载。