Geek asks and answers

Iterate over large sets in C #: very long

I recently started a WPF application. I connected this to BaseX (XML based) and got about a million records. I wanted to iterate over the records, compute something for each record, and then write this back to the database:

IEnumerable<Result> resultSet = baseXClient.Query("...", "database");
foreach (Result result in resultSet) 
{
    ...
}

Problem: The inner part of foreach is never reached. the Query () method returns pretty quickly, but when foreach is reached, C # seems to do SOMETHING with the collection, the code doesn't last very long (at least 10 minutes, never let it work anymore). What's going on here? I tried to limit the number of items received. When you get 100,000 results, the same thing happens, but the code continues in about 10-20 seconds. With a full million results, C # seems to be stuck forever ...

Any ideas? considers

Edit: why this happens As some of you have pointed out, the reason for this behavior is that the request is actually evaluated only when called MoveNext()in Enumerator inside Enumerable. My database doesn't seem to be able to return one value at a time, but instead returns just one million data sets at once. I will try to switch to another database (if possible, Apache Lucene, since it has good support for full-text search) and edit this message to let it know if it has changed anything.
PS: Yes, I know that a million results are many. It is not intended for real-time use; it is just a step for preparing the data. Although I did not expect the code to start in a few seconds, I was still surprised to see SUCH low performance in the database.

: . XML Apache Lucine. ! , Lucine - , , . , - !

+5
5

- ... , , , , . , , , , , , foreach.

:

"foreach" - , IEnumerable IEnumerator: IEnumerable.GetEnumerator, IEnumerator.MoveNext.

GetEnumerator ( LINQ) ( .

MoveNext , , . LINQ .

+3

- , rsultSet, ( ) , , .

XML.

+5

foreach ( ), . , , , , .

, , 1 . ( , , )

+2

foreach, ToList . ( , , )

0

Have you tried a more traditional loop to see if this works?

IEnumerable<Result> resultSet = baseXClient.Query("...", "database");
for (int x =0; x < resultSet.Count; x++)
{
    Result result = resultSet[x];
    ...
}
-1
source

More articles:


All Articles