Using Sqoop2 to import mysql data to HDFS

云中漫步

　　On Sqoop2 server:
　　MySQL JDBC Driver:
　　cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/mysql-connector-java.jar
　　Oracle JDBC Driver:
　　cp ojdbc6.jar /var/lib/sqoop2/oracle-connector-java.jar
　　Microsoft SQL Server JDBC Driver:
　　cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/
　　cd /var/lib/sqoop2
　　chown sqoop2:sqoop2 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
　　chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
　　restart sqoop2 service
　　On Sqoop2 client:
　　MySQL JDBC Driver:
　　cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop2/mysql-connector-java.jar
　　Oracle JDBC Driver:
　　cp ojdbc6.jar /var/lib/sqoop2/oracle-connector-java.jar
　　Microsoft SQL Server JDBC Driver:
　　cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop2/
　　cd /var/lib/sqoop2
　　chown sqoop2:sqoop2 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
　　chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
　　(
　　For using Sqoop 1 client, you should add sqoop 1 client  service to client node as gateway role, then put JDBC Driver to /var/lib/sqoop
　　MySQL JDBC Driver:
　　cp mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar /var/lib/sqoop/mysql-connector-java.jar
　　Oracle JDBC Driver:
　　cp ojdbc6.jar /var/lib/sqoop/oracle-connector-java.jar
　　Microsoft SQL Server JDBC Driver:
　　cp sqljdbc_4.0/enu/sqljdbc4.jar /var/lib/sqoop/
　　cd /var/lib/sqoop
　　chown sqoop:sqoop mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
　　chmod 755 mysql-connector-java.jar oracle-connector-java.jar sqljdbc4.jar
　　sudo -u hdfs hadoop fs -mkdir /user/sqoop
　　sudo -u hdfs hadoop fs -chown sqoop:sqoop /user/sqoop
　　sqoop list-databases --connect jdbc:mysql://mysql-server-ip --username username --password
　　sqoop list-tables --connect jdbc:mysql://mysql-server-ip/databasename --username username --password
　　)
　　sqoop2 (you can also use HUE web UI > sqoop transfer)
　　sqoop:000> set server --host sqoop2-server-ip
　　sqoop:000> show connector --all
　　sqoop:000> create connection --cid 1

　　Creating connection for connector with>　　Please fill following values to create new connection object
　　Name: mysql-conection
　　Connection configuration

　　JDBC Driver>　　JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname
　　Username: username
　　Password: *******
　　JDBC Connection Properties:
　　There are currently 0 values in the map:
　　entry#

　　Security>　　Max connections: 10

　　New connection was successfully created with validation status FINE and persistent>　　sqoop:000> create job --xid 1 --type import

　　Creating job for connection with>　　Please fill following values to create new job object
　　Name: export-job1
　　Database configuration
　　Schema name: dbname
　　Table name: tablename
　　Table SQL statement:
　　Table column names:
　　Partition column name:
　　Nulls in partition column:
　　Boundary query:
　　Output configuration
　　Storage type:
　　0 : HDFS
　　Choose: 0
　　Output format:
　　0 : TEXT_FILE
　　1 : SEQUENCE_FILE
　　Choose: 0
　　Compression format:
　　0 : NONE
　　1 : DEFAULT
　　2 : DEFLATE
　　3 : GZIP
　　4 : BZIP2
　　5 : LZO
　　6 : LZ4
　　7 : SNAPPY
　　Choose: 0
　　Output directory: /user/sqoop2/output
　　Throttling resources
　　Extractors:
　　Loaders:

　　New job was successfully created with validation status FINE  and persistent>　　sqoop:000> start job --jid 1
　　sqoop:000> status job --jid 1

　　JDBC Driver>　　JDBC Connection String: jdbc:mysql://remote-mysql-server/dbname

　　JDBC Driver>　　JDBC Connection String: jdbc:sqlserver://remote-mysql-server/dbname

　　JDBC Driver>　　JDBC Connection String: