Geek asks and answers

Oozie + Sqoop: JDBC Driver Jar Location

I have a 6 node hadouop clouder based cloudera and am trying to connect to oracle database from sqoop action in oozie.

I copied my ojdbc6.jar file to the sqoop lib location (for me this happened: /opt/cloudera/parcels/CDH-4.2.0-1.cdh4.2.0.p0.10/lib/sqoop/lib/) at all nodes and checked that I can run a simple sqoop eval from all 6 nodes.

Now, when I run the same command using the Oozie sqoop action, I get "Can't load db driver class: oracle.jdbc.OracleDriver"

I read this article about using shared libraries, and it makes sense to me when we talk about my dependency dependencies task / action / workflow. But I see the JDBC driver installation as an extension for sqoop, and therefore I think it belongs to the sqoop installation library.

Now the question is, what sqoop sees this ojdbc6 box that I put the lib folder in it, where does my Oozie workflow not see it?

Is this something expected or am I missing something?

As an aside, what do you think about where is the right place for the JDBC jar driver?

Thanks in advance!

+5
source share
5 answers

JDBC- ( , ) sharelib Oozie HDFS. Hortonworks Data Platform 1.2 Cloudera 4.2, , JDBC /user/oozie/share/lib/sqoop. Sqoop JDBC Oozie.

jQBC- jar sqoop lib . setupt sqoop eval . , , . , JDBC- HDFS, , . . , , sharelib , .

+6

CDH5 '/user/oozie/share/lib/lib_ ${timestamp}/sqoop', sharelib oozie.

sharelib:

oozie admin -oozie http://localhost:11000/oozie -sharelibupdate

+3

CDH-5, jar- JDBC ( , ) '/user/oozie/share/lib/lib_timestamp/sqoop' HDFS.

+1

, mysql jar. cloudera 4.4 oozie admin -oozie http://localhost:11000/oozie -sharelibupdate

, :

  • Hue hdfs

  • Hue UI jar /user/oozie/share/lib/sqoop hdfs :

hadoop put /var/lib/sqoop2/mysql-connector-java.jar /user/oozie/share/lib/sqoop

  1. jar oozie.
0

Oozie ShareLib. :

Shell

sudo -u hdfs hadoop fs -chown cloudera:cloudera /user/oozie/share/lib/lib_20170719053712/sqoop
hdfs dfs -put /var/lib/sqoop/mysql-connector-java.jar /user/oozie/share/lib/lib_20170719053712/sqoop
sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie/share/lib/lib_20170719053712/sqoop

oozie admin -oozie http://localhost:11000/oozie -sharelibupdate
oozie admin -oozie http://localhost:11000/oozie -shareliblist sqoop

Hue Sqoop Client

sqoop list-tables --connect jdbc:mysql://localhost/retail_db --username root --password cloudera

More details at:

https://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/

0
source

More articles:


All Articles