Geek asks and answers

Hadoop MapReduce chain using Pipes (C ++)

Does anyone know how to associate two MapReduce APIs with Pipes APIs? I already link the two MapReduce in the previous project with JAVA, but today I need to use C ++. Unfortunately, I have not seen examples in C ++.

Has anyone done this already? It's impossible?

+5
source share
2 answers

Finally, I can do the work of Hadoop Pipes. Here are a few steps to create wordcount examples available in src / examples / pipes / impl /.

I have a working Hadoop 1.0.4 cluster configured according to the instructions described in the documentation.

Pipes, , . ++ 32- 64- . , :

# cd /src/c++/utils
# ./configure
# make install

# cd /src/c++/pipes
# ./configure
# make install

/src/++, .

, −lssl −lcrypto . . wordcount, src/examples/pipes/impl/.

wordcount-nopipe . . ://. , InputFormat. , :

# bin/hadoop pipes −D hadoop.pipes.java.recordreader=false −D hadoop.pipes.java.recordwriter=false −libjars hadoop−1.0.4/build/hadoop−test−1.0.4.jar −inputformat org.apache.hadoop.mapred.pipes.WordCountInputFormat −input file:///input/file −output file:///tmp/output −program wordcount−nopipe

, org.apache.hadoop.mapred.pipes.Submitter.java 1.0.4, java-, InputFormat. , setIsJavaRecordReader(job,true);, , (http://web.archiveorange.com/archive/v/RNVYmvP08OiqufSh0cjR).

if(results.hasOption("−inputformat")) { 
    setIsJavaRecordReader(job, true);
    job.setInputFormat(getClass(results, "−inputformat", job,InputFormat.class));
}
0

Oozie . Pipes MapReduce.

+1

More articles:


All Articles