Geek asks and answers

Reading a directory using an api file: how to handle a directory containing 20000-30000 files?

I am in the process of writing a simple java program that reads the contents of a directory and print the file names and last modified time.

The problem I'm looking at is the repository I'm reading is quite huge, and there are some cases where files in the same directory can exceed 20,000. Using the api file

    `file.listFiles()`

will start the creation of 20,000 file objects, my problem in this may slow down the process, memory may also swell

Is there a way to do ie to tell java to scan the directory 50 files at a time or at least repeat one file at a time, instead of loading all the objects in memory at once.

+3
source share
2 answers

You must use java.nio.file.FileVisitor via java.nio.file.Files.walkFileTree (...). It was introduced in Java 7 specifically for this use case.

+5
source

I would not worry about memory; I would worry about the disk access time for these files.

File access time can be 8 ms, even for SSD it can be 100 microseconds. The time taken to create the File object may be less than a microsecond.

You are right that 20,000 File objects are not trivial, but if this is your main problem, you are doing very well.

Instead, you can use list()one that will be slightly more memory efficient and create the file only as needed.

+1
source

More articles:


All Articles