Geek asks and answers

Writing to a file in S3 from jar to EMR on AWS

Is there a way that I can write a file from my Java-jar to the S3 folder where my reduction files will be written? I tried something like:

    FileSystem fs = FileSystem.get(conf);
    FSDataOutputStream FS = fs.create(new Path("S3 folder output path"+"//Result.txt"));        

    PrintWriter writer  = new PrintWriter(FS);
    writer.write(averageDelay.toString());
    writer.close();
    FS.close();

Here Result.txt is the new file I would like to write.

+3
source share
3 answers

Answering my own question: -

I found my mistake. I need to pass the URI of the path of the S3 folder to the fileSystem object, as shown below: -

 FileSystem fileSystem = FileSystem.get(URI.create(otherArgs[1]),conf);
 FSDataOutputStream fsDataOutputStream = fileSystem.create(new Path(otherArgs[1]+"//Result.txt"));      
 PrintWriter writer  = new PrintWriter(fsDataOutputStream);
 writer.write("\n Average Delay:"+averageDelay);
 writer.close();
 fsDataOutputStream.close();
0
source 
FileSystem fileSystem = FileSystem.get(URI.create(otherArgs[1]),new JobConf(<Your_Class_Name_here>.class));
FSDataOutputStream fsDataOutputStream = fileSystem.create(new     
Path(otherArgs[1]+"//Result.txt"));      
PrintWriter writer  = new PrintWriter(fsDataOutputStream);
writer.write("\n Average Delay:"+averageDelay);
writer.close();
fsDataOutputStream.close();

This is how I processed the conf variable in the above code block, and it worked like a charm.

0
source

Here is another way to do this in Java using AWS S3 putObject directly with a string buffer.

... AmazonS3 s3Client;

public void reduce(Text key, java.lang.Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context) throws Exception {

    UUID fileUUID = UUID.randomUUID();
    SimpleDateFormat sdf = new SimpleDateFormat("yyy-MM-dd");
    sdf.setTimeZone(TimeZone.getTimeZone("UTC"));

    String fileName = String.format("nightly-dump/%s/%s-%s",sdf.format(new Date()), key, fileUUID);
    log.info("Filename = [{}]", fileName);

    String content = "";
    int count = 0;
    for (Text value : values) {
        count++;
        String s3Line = value.toString();
        content += s3Line + "\n";
    }
    log.info("Count = {}, S3Lines = \n{}", count, content);


    PutObjectResult putObjectResult = s3Client.putObject(S3_BUCKETNAME, fileName, content);
    log.info("Put versionId = {}", putObjectResult.getVersionId());

    reduceWriteContext("1", "1");

    context.setStatus("COMPLETED");
}
0
source

More articles:


All Articles