Geek asks and answers

How to make a parallel iterator over some source?

I would like to have an iterator that can be read by multiple threads at the same time so that I can process the iterator source data in parallel. The problem is that I cannot really combine hasNext()with its logical next(), as they can go to different threads. (That is, two threads can cause hasNext(), each sees true, and then the second thread fails because there is only one element.) My problem is that for some sources I really don't know if it has the next element until I I will try to read it. One such example is reading lines from a file; another reads instances Termfrom the Lucene index.

I was thinking about setting up a queue inside an iterator and feeding the queue as a separate thread. Thus, it hasNext()is implemented in terms of queue size. But I do not understand how I can guarantee that the queue is full, because this thread can become hungry.

Should I ignore the contract with Iterator and just call next()exhaustively until it is thrown away NoSuchElementException?

Is there a more elegant way to solve the problem?

+3
source share
4 answers

Can your threads just pull from a BlockingQueue instead of an Iterator. As you discovered, Iterators are not very suitable for simultaneous access.

LinkedBlockingQueue, queue.poll(), .

+6

/ , ( ) NoSuchElementExceptions: iterator.next() " ", , , . , true hasNext(), , ( ).

, . .

, NoSuchElementException hasNext(), - .

+1

The selected answer will work, but it introduces complexity and potential unnecessary buffering. Why not ignore the Iterators contract and write your own:

public interface ConcurrentIterator<T> {

    T next() throws EndOfIterationException;

}

It will be thread safe if your implementation. You can even wrap Iteratorin it.

0
source

I missed the point, but could you use a synchronized block in this situation?

synchronized(iterator)
{
    if (iterator.hasNext()) element = iterator.next();
}

Here, when one thread uses an iterator, no other thread can access it.

-1
source

More articles:


All Articles