Geek asks and answers

How to set classpath for java program in hasoop file system

I'm trying to figure out how to set a class path that references HDFS? I can not find any links.

 java -cp "how to reference to HDFS?" com.MyProgram

If I cannot reference the hadoop file system, then I need to copy all the links to third-party libraries / baths somewhere under $ HADOOP_HOME on each hadoop machine ... but I want to avoid this by placing the files in the hadoop file system. Is it possible?

An example hasoop command line for starting a program (my expectation seems to be, maybe I'm wrong):

hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.0.3.jar -input inputfileDir -output outputfileDir -mapper /home/nanshi/myprog.java -reducer NONE -file /home/nanshi/myprog.java

However, on the command line above, how to add a java classpath? like -cp "/home/nanshi/wiki/Lucene/lib/lucene-core-3.6.0.jar:/home/nanshi/Lucene/bin"

+5
source share
3 answers

, , . , .

1). , , - $HADOOP_HOME/lib (,/usr/local/hadoop-0.22.0/lib) worktracker tasktracker.

2) libjars .   hadoop jar -libjars comma_seperated_jars

3) lib . , .

4) .

5). , .

+11

classpath HDFS. Java- - :

hdfs://path/to/your/file

, , -libjars. , ( ), -libjars . , , GenericParser ( -libjars) JobClient , , , .

, MR hasoop script, bin/ .

( , ):

 hadoop jar jarfilename.jar DriverClassInTheJar 
 -libjars comma-separated-list-of-libs <input> <output>
+2

You can specify the jar path as
-libjars hdfs: // namenode / path_to_jar, I used this with Hive.

+2
source

More articles:


All Articles