Geek asks and answers

Parallel class for loading pages

I have code with a parallel class for loading pages from the Internet. Since I load about 3,000 pages, I want to know if this is the best way.

Parallel.For(0, 3000, i =>
            {
                Console.WriteLine(i.ToString());
                //HttpDownloader is my class for downloading
                HttpDownloader ht = new HttpDownloader(s[i]);
                string a = ht.GetPage();
                Console.WriteLine(i.ToString());
            });

After that I run 2 func: pharsing (string html) and save () // Save to DB How can I do this in Parallel ?? And, if I want it to start the background, do I need to paste it into BackgroundWorker?

+3
source share
1 answer

A parallel task library (TPL, from where Parallel) is a way to go - as you already do. But you can make it more clear using Parallel.ForEachover Parallel.For:

var urls = new List<string> { "http://google.com", "http://yahoo.com" };

Parallel.ForEach(urls, url => {
    using (var client = new WebClient())
    {
        var contents = client.DownloadString(url);
        // parse contents
        // add to database
    }
});

, , , ( ), : http://blogs.msdn.com/b/pfxteam/archive/2009/08/04/9857477.aspx

, , .

+3

More articles:


All Articles