Geek asks and answers

Example shortest paths for joystick ClassNotFoundException

I am trying to run an example of the shortest paths from a gyroscopic incubator ( https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example ). However, instead of executing the example from giraph - * - dependencies.jar, I created my own work. When I created one Job file, presented in the example, I got

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.test.giraph.Test$SimpleShortestPathsVertexInputFormat

Then I moved the inner classes (SimpleShortestPathsVertexInputFormat and SimpleShortestPathsVertexOutputFormat) to separate the files and renamed them just in case (SimpleShortestPathsVertexInputFormat_v2, SimpleShortestPathsVertexOutputFormat_v2); classes are no longer static . This solved the problems of a class not found for SimpleShortestPathsVertexInputFormat_v2, however I still get the same error for SimpleShortestPathsVertexOutputFormat_v2. The stack trace is shown below.

INFO mapred.JobClient: Running job: job_201205221101_0003
INFO mapred.JobClient:  map 0% reduce 0%
INFO mapred.JobClient: Task Id : attempt_201205221101_0003_m_000005_0, Status : FAILED
    java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.test.giraph.utils.SimpleShortestPathsVertexOutputFormat_v2
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
            at org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134)
            at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
            at org.apache.hadoop.mapred.Task.initialize(Task.java:490)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:415)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
            at org.apache.hadoop.mapred.Child.main(Child.java:253)
    Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.test.giraph.utils.SimpleShortestPathsVertexOutputFormat_v2
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:890)
            ... 9 more

I checked my work and all classes are there. In addition, I use the 0.20.203 chaos in pseudo-distributed mode. The following is a way to get my work started.

hadoop jar giraphJobs.jar org.test.giraph.Test -libjars /path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar /path/to/input /path/to/output 0 3

HADOOP_CLASSPATH giraph - * - dependencies.jar. PageRankBenchmark ( giraph - * - dependencies.jar), shortes ( giraph - * - dependencies.jar). hadoop (- , , "" ). - ? .

(, , )

, Job-libjars ( HADOOP_CLASSPATH, ). .

hadoop jar giraphJobs.jar org.test.giraph.Test -libjars /path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar,/path/to/job.jar /path/to/input /path/to/output 0 3

. . , jar "classpath"? - , ? ( ) , "classpath". .

+5
1

. run() :

...
@Override
public int run(String[] argArray) throws Exception {
    Preconditions.checkArgument(argArray.length == 4,
        "run: Must have 4 arguments <input path> <output path> " +
        "<source vertex id> <# of workers>");

    GiraphJob job = new GiraphJob(getConf(), getClass().getName());
    // This is the addition - it will make hadoop look for other classes in the same     jar that contains this class
    job.getInternalJob().setJarByClass(getClass());
    job.setVertexClass(getClass());
    ...
}

setJarByClass() hadoop , , getClass(), jar -libjars.

+3

More articles:


All Articles