Hadoop Eclipse plug-in (installed in Eclipse IDE or Spring Tool Suite) eases the experience of Map/Reduce on Hadoop. Hadoop distribution does not include Hadoop Eclipse plug-in jar, but includes source code of that plug-in.
In this post, we'll see all the steps to build Hadoop eclipse plug-in from source code and install that plug-in in Eclipse / Spring Tool Suite to access Hadoop Environment in "Map/Reduce" perspective of the IDE.
Tools and Technologies used in this article :
Hadoop 1.2.0
Ant
Spring Tool Suite (STS) 3.2.0
Navigate to /src/contrib/eclipse-plugin. Following highlighted files will be modified.
1. Try Ant Build
Lets us first see, what happens if we try to build Hadoop Eclipse plug-in from the provided source code. Terminal
static-217:hadoop srccodes$ cd /Users/srccodes/hadoop/src/contrib/eclipse-plugin
static-217:eclipse-plugin srccodes$ ant jar
Buildfile: /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml
ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: /Users/srccodes/hadoop/ivy/ivy-2.1.0.jar
[get] Not modified - so not downloaded
ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
compile:
jar:
BUILD SUCCESSFUL
Total time: 2 seconds
Above echo message indicates that eclipse.home is not set and build of the plugin is skipped.
2. Modify build.properties
Open build.properties in a text editor and set eclipse.home with your Eclipse / STS installation directory. Also define version of different jars required by Hadoop Eclipse plug-in. build.properties
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# set eclipse installation path
eclipse.home=/Users/srccodes/software/springsource/sts-3.2.0.RELEASE
# version of different jars required by Hadoop eclipse plugin
version=1.2.0
commons-cli.version=1.2
commons-configuration.version=1.6
commons-httpclient.version=3.0.1
commons-lang.version=2.4
jackson-core-asl.version=1.8.8
jackson-mapper-asl.version=1.8.8
3. Try Ant Build Once Again
Now eclipse.home is set, but we are not done yet. To find out why, try ant build once again. This time we will get java compilation errors as shown below. Terminal
static-166:eclipse-plugin srccodes$ ant jar
Buildfile: /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml
check-contrib:
init:
[echo] contrib: eclipse-plugin
init-contrib:
ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: /Users/srccodes/hadoop/ivy/ivy-2.1.0.jar
[get] Not modified - so not downloaded
ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
compile:
[echo] contrib: eclipse-plugin
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml:62: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 45 source files to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/classes
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:28: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.FileStatus;
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:29: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.Path;
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSPath.java:25: package org.apache.hadoop.hdfs does not exist
[javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
[javac] ^
:
:
:
:
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopServer.java:291: cannot find symbol
[javac] symbol : class Configuration
[javac] location: class org.apache.hadoop.eclipse.server.HadoopServer
[javac] public Configuration getConfiguration() {
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/ConfProp.java:24: package org.apache.hadoop.conf does not exist
[javac] import org.apache.hadoop.conf.Configuration;
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopServer.java:468: cannot find symbol
[javac] symbol : class FileSystem
[javac] location: class org.apache.hadoop.eclipse.server.HadoopServer
[javac] public FileSystem getDFS() throws IOException {
[javac] ^
:
:
:
:
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopJob.java:302: package Counters does not exist
[javac] Counters.Group group = counters.getGroup(groupName);
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopJob.java:305: package Counters does not exist
[javac] for (Counters.Counter counter : group) {
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFile.java:74: cannot find symbol
[javac] symbol : class FileStatus
[javac] location: class org.apache.hadoop.eclipse.dfs.DFSFile
[javac] FileStatus fs = getDFS().getFileStatus(path);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors
BUILD FAILED
/Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml:62: Compile failed; see the compiler error output for details.
Total time: 3 seconds
4. Modify build.xml
Open build.xml and modify path (id="classpath") and target (name="jar") as highlighted below. build.xml
Note : Hadoop Eclipse Plug-in code is dependent on Hadoop jars. So it is required to add those in classpath of 'compile' target. Also we need to include jars required by the eclipse plugin in 'jar' target. 5. Modify MANIFEST.MF
Open MANIFEST.MF in text editor and modify Bundle-ClassPath as highlighted below to incorporate the changes made in build.xml. META-INF/MANIFEST.MF
6. Build Hadoop Eclipse Plug-in
Execute ant jar or ant clean package. On successful build, Hadoop Eclipse Plug-in jar (hadoop-eclipse-plugin-1.2.0.jar) will be created inside /build/contrib/eclipse-plugin/ Terminal
static-217:eclipse-plugin srccodes$ ant jar
Buildfile: /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml
check-contrib:
init:
[echo] contrib: eclipse-plugin
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/classes
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/test
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/system
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/system/classes
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/examples
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/test/logs
init-contrib:
ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: /Users/srccodes/hadoop/ivy/ivy-2.1.0.jar
[get] Not modified - so not downloaded
ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
compile:
[echo] contrib: eclipse-plugin
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml:64: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 45 source files to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/classes
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
jar:
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/hadoop-core-1.2.0.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/hadoop-core.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-cli-1.2.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-cli-1.2.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-configuration-1.6.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-configuration-1.6.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-httpclient-3.0.1.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-httpclient-3.0.1.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-lang-2.4.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-lang-2.4.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/jackson-core-asl-1.8.8.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/jackson-core-asl-1.8.8.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/jackson-mapper-asl-1.8.8.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/jackson-mapper-asl-1.8.8.jar
[jar] Building jar: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-1.2.0.jar
BUILD SUCCESSFUL
Total time: 4 seconds
7. Install Hadoop Eclipse Plug-in
Copy Hadoop Eclipse Plug-in jar (hadoop-eclipse-plugin-1.2.0.jar) and paste it to /plugins directory. Start / Restart (if already running) Eclipse / Spring Tool Suite IDE.
8. Check Hadoop Eclipse Plug-in
Navigate Window --> Open Perspective --> Other and select Map/ReduceOpen perspective. You'll be able to seeMap/Reduce Locations window where New Hadoop Location can be added.
Now, Eclipse environment for Hadoop is ready for you to explore.
QQ群⑧：
窥视卡
雷达卡
发表于 2015-7-14 10:44:50
Now, Eclipse environment for Hadoop is ready for you to explore.
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡