Geek asks and answers

Is it possible to run hasoop fs -getmerge in S3?

I have work to cut an elastic card that writes some files to S3, and I want to combine all the files to create a unique text file.

Currently, I manually copy the folder with all the files to our HDFS (hasoop fs copyFromLocal), then I run hasoop fs -getmerge and hasoop fs copyToLocal to get the file.

Anyway, should you use hasoop fs directly on S3?

+5
source share
3 answers

Actually, this answer about getmerge is incorrect. getmerge expects a local destination and will not work with S3. It throws out IOExceptionif you try to answer using -getmerge: Wrong FS :.

Using:

hadoop fs [generic options] -getmerge [-nl] <src> <localdst>
+1

(, , -), :

  • ()

    hadoop fs -getmerge hdfs://[FILE] [LOCAL FILE]

  • S3, ()

    hadoop dfs -moveFromLocal [LOCAL FILE] s3n://bucket/key/of/file
+1

getmerge, hadoop fs EMR S3, HDFS. , SSH node :

hadoop fs -ls s3://<my_bucket>/<my_dir>/

S3 .

, hadoop fs -getmerge . , S3 ( s3://) HDFS-.

0
source

More articles:


All Articles