Geek asks and answers

Best way to write Big JSON files avoiding problems with OutOfMemory

First, note that today is my first day with GSON. I am trying to write a Json file using a library GSON. I have thousands JsonObjectsinside ArrayList. When the Json file is written, it should look something like this.

[
    {
        "hash_index": "00102x05h06l0aj0dw",
        "body": "Who signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 0,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    },
    {
        "hash_index": "00102x05h06l0aj0dw0iz0kn0l@0t#0",
        "body": "Who signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 1,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    }
]

I am currently writing JSOn using the code below.

 private void writeNewJsonFile() throws IOException
    {
        System.out.println("Starting to write the JSON File");
        //Add everything into a JSONArray
        JsonArray jsonArrayNew = new JsonArray();

        for(int i=0;i<jsonObjectHolder.size();i++)
        {
            System.out.println("inside array");
            jsonArrayNew.add(jsonObjectHolder.get(i));
        }


        //Write it to the File
    /*  File file= new File("items_Articles_4_1.json");

        FileWriter fw = new FileWriter(file);;
        fw.write(jsonArrayNew.toString());
        fw.flush();
        fw.close();*/

        System.out.println("outside array");

        ByteArrayInputStream input = new ByteArrayInputStream(jsonArrayNew.toString().getBytes());

        Long contentLength = Long.valueOf(jsonArrayNew.toString().getBytes().length);

        ObjectMetadata metaData = new ObjectMetadata();
        metaData.setContentLength(contentLength);

        s3.putObject(outputBucket,outputFile,input,metaData);


    }

Here I convert JsonArrayto Stringand write. I have a fear that this will crash soon with Big Json arrays and give it to me OutOfMemoryException. Just as I read part of Json in parts using GSON, is there any way to write a Json file in parts or something else that can avoid the problems OutOfMemoryException?

+3
source share
1

:

WriteJsonArrayByParts<Cache> write = new WriteJsonArrayByParts<Cache>(fileNameTest, " ");
write.writeStart();
for(Cache cache : listOfObjects()) {
    write.writeObject(cache, Cache.class);
}
write.writeEnd();
write.close();

...

public static class WriteJsonArrayByParts<T> {
    Gson gson = new Gson();
    JsonWriter writer;

    public WriteJsonArrayByParts(String fileNameWithPath, String indent) throws Exception {
        OutputStream os = new FileOutputStream(fileNameWithPath, false);
        BufferedOutputStream osb = new BufferedOutputStream(os, 8 * 1024);

        writer = new JsonWriter(new OutputStreamWriter(osb, StringUtil.UTF_8));
        writer.setIndent(indent);
    }

    public void writeStart() throws IOException {
        writer.beginArray();
    }

    @SuppressWarnings("unchecked")
    public void writeObject(T t, Class<?> resultClass) throws IOException {
        ((TypeAdapter<Object>) gson.getAdapter(resultClass)).write(writer, t);
    }

    public void writeEnd() throws IOException {
        writer.endArray();
    }

    public void close() throws IOException {
        writer.close();
    }
}
0

More articles:


All Articles