Geek asks and answers

Java: reading at the same time on an InputStream

For a while I looked around and I am a little confused by this problem. I want to be able to receive an input stream and read it simultaneously in segments. Segments do not interact with each other, but represent only the values ​​that need to be inserted or updated in the database from the downloaded file. Is it possible to read the input stream at the same time, setting the segment size, and then just skip ahead before unscrewing the new stream to process the conversion and insert / update?

In fact, the file is a list of IDs (one identifier per line), although it would be preferable if I could specify a separator. Some files can be huge, so I would like to process and convert the data into segments, so that after inserting / updating the database, I can free up the JVM memory. Is it possible? And if there are any libraries that already do this?

Greetings and thanks in advance,

Alexey Blue.

+5
source share
3 answers

, , , . , , , , , .

, System.in . , , 1000 , , .

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Main {
    public static class Worker implements Runnable {
        private final String line;

        public Worker(String line) {
            this.line = line;
        }

        @Override
        public void run() {
            // Process line here.
            System.out.println("Processing line: " + line);
        }
    }

    public static void main(String[] args) throws IOException {
        // Create worker thread pool.
        ExecutorService service = Executors.newFixedThreadPool(4);

        BufferedReader buffer = new BufferedReader(new InputStreamReader(System.in));
        String line;

        // Read each line and hand it off to a worker thread for processing.
        while ((line = buffer.readLine()) != null) {
            service.execute(new Worker(line));
        }
    }
}
+6

, InputStream. read, reset mark - , , , .

, . skip() , . BufferedReader , .

+1

, , , , . Java RandomAccessFile java.in SeekableByteChannel java.nio:

Java

http://docs.oracle.com/javase/tutorial/essential/io/rafs.html

, java.nio. Java NIO FileChannel / FileOutputstream

, , . , . , , . , .

Now you know how to read the same file at the same time with many different offsets.

But think about performance. Despite the number of threads, you only have one drive and random reads (many access threads to the same file), performance is much slower than sequential reads (one thread reads one file). Even if it raids 0 or 1 - it does not matter. Sequential reading is always much faster. Therefore, in your case, I would advise you to read the file in one stream and pass data from this read stream to other streams.

+1
source

More articles:


All Articles