Geek asks and answers

Using threadpools / threading to read large txt files?

In the previous question I posted:

I need to read some very large txt files and use either multiple threads or one thread to do this depending on user input. Let's say I have a main method that receives user input, and the user requests one stream and wants to process 20 txt files for that stream. How would I do that? Please note that the following is not my code or its setting, but only what is an “idea”.

Example:

int numFiles = 20;
int threads = 1;

 String[] list = new String[20];
 for(int i = 1; i < 21; i++){
   list[i] = "hello" + i + ".txt";//so the list is a hello1.txt, hello2.txt, ...,  hello20.txt
 }

 public void run(){
 //processes txt file
 }

So how can I do this with a single thread? With 20 threads?

And the user suggested using threadPools:

, , , . Java Executors.newFixedThreadPool factory . IBM Java.

, sortAndMap (String x), txt , ,

Executors.newFixedThreadPool(numThreads);

threadPools, ?

+3
3

, , .

, , , , SSD, . , : , .

, . , , , .

, ( ) - ( ). - , , , .

- :

int numFiles = 20;
int threads = 4;

ExecutorService exec = Executors.newFixedThreadPool(threads);

for(int i = 0; i < numFiles; i++){
    String[] fileContents = // read current file;
    exec.submit(new ThreadTask(fileContents));
}

exec.shutdown();
exec.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
...

class ThreadTask implements Runnable {

   private String[] fileContents;

   public ThreadTask(String[] fileContents) {
        this.fileContents = fileContents;
   }

   public void run(){
      //processes txt file
   }
}
+11

tutorial concurrency. concurrency , , .

+1

Thus, the call newFixedThreadPool()returns an instance of ExecutorService . You can reference the JavaDoc, which is pretty comprehensive and provides an effective example. You will want either a number submitor a invokeAllnumber Callablethat performs your file processing tasks, resulting in several Future. Their methods get()will give you the result of the task after completion (you must write this part yourself :))

+1
source

More articles:


All Articles