Geek asks and answers

Parallel Process - Intensive I / O Function

I have this sample code.

List<Dictionary<string,string>> objects = new List<Dictionary<string,string>>();

foreach (string url in urls)
{
    objects.add(processUrl(url))
}

I need to process the url, processUrlload the page down and run a lot of regular expressions to extract some information and return a “C # JSON like” object, so I want to run this in parallel, and in the end I need a list of objects, so I need to wait all tasks to continue the process, how can I do this? I use many examples, but no one keeps returning.

Hi

+3
source share
3 answers

Like this?

var results = urls.AsParallel().Select(processUrl).ToList();

C Parallel:

Parallel.ForEach(
    urls, 
    url =>
    {
        var result = processUrl(url);
        lock (syncOjbect)
            objects.Add(result);
    };

or

var objects = new ConcurrentBag<Dictionary<string,string>>();
Parallel.ForEach(urls, url => objects.Add(processUrl(url)));
var result = objects.ToList();

or with tasks:

var tasks = urls
    .Select(url => Task.Factory.StartNew(() => processUrl(url)))
    .ToArray();

Task.WaitAll(tasks);
var restuls = tasks.Select(arg => arg.Result).ToList();
+2
source

Refactoring first as

processUrl(url, objects);

and make the task responsible for adding the results to the list.

, .

: async .NET .

0

You can use PLinq extensions, this requires .NET 4.0

System.Threading.Tasks.Parallel
          .ForEach(urls, url => {
             var result = processUrl(url);
             lock(objects)
             {
                  objects.Add(result);
             }
           });
-1
source

More articles:


All Articles