Geek asks and answers

C # Multithreaded application - structure?

So, I will make an application for checking links, if they are available (live). My question is how to make threads always busy. What I mean: An application starts 100 threads (e.g. with a FOR loop) with 100 different URLs. Therefore, when 1 of the threads exits (check if the URL is available) to get a new URL and immediately start again. Thus, 100 threads will work non-stop until all URLs are verified.

How can i do this?

+5
source share
3 answers

What you are looking for is called a Producer-Consumer Model . You have a resource pool that contains a list of URLs to check, one thread can fill this pool, and your conumer threads can pull out of this pool if you have .NET 4 Parallel.ForEach does most of the work for you.

Using 100 threads will also most likely not be the optimal number of threads, just let the parallel task library manage the number of threads for you.

Here is an example if the list is pre-populated and no more items are added when the stream starts.

//Parallel.Foreach will block until it is done so you may want to run this function on a background worker.
public void StartThreads()
{
    List<string> myListOfUrls = GetUrls();

    Parallel.Foreach(myListOfUrls, ProcessUrl);
}


private void ProcessUrl(string url)
{
    //Do your work here, this code will be run from multiple threads.
}

If you need to fill the collection when it starts, replace it List<string>with a parallel collection, for example BlockingCollection

BlockingCollection<string> myListOfUrls = new BlockingCollection();

//Parallel.Foreach will block until it is done so you may want to run this function on a background worker.
public void StartThreads()
{
    if(myListOfUrls.IsComplete == true)
    {
        //The collection has emptied itself and you told it you where done using it, you will either need to throw a exception or make a new collection.
        //use IsCompleatedAdding to check to see if you told it that you are done with it, but there still may be members left to process.
        throw new InvalidOperationException();
    }

    //We create a Partitioner to remove the buffering behavior of Parallel.ForEach, this gives better performance with a BlockingCollection.
    var partitioner = Partitioner.Create(myListOfUrls.GetConsumingEnumerable(), EnumerablePartitionerOptions.NoBuffering);
    Parallel.ForEach(partitioner, ProcessUrl);
}

public void StopThreads()
{
    myListOfUrls.CompletedAdding()
}

public void AddUrl(string url)
{
    myListOfUrls.Add(url);
}

private void ProcessUrl(string url)
{
    //Do your work here, this code will be run from multiple threads.
}

, , , .

, / 100 , : Core 2GB RAM XP Parallel.Foreach 5 ( ThreadPool.SetMinThreads) 100 30-40% . Parallel.Foreach. PS: WebClient wc = new WebClient(); var s = wc.DownloadString(url); ( google) - L.B

+10

Parallel CTP, foreach , .

Google - .

, 100 , , , .

+2

You can use ThreadPooland provide it with a list of URLs to process, and then let the method DoWorkhandle the check whether they are live or not, for example.

 foreach (string s in URLs)
 {
       ThreadPool.QueueUserWorkItem(new WaitCallback(DoWork), s);
 }

 public void DoWork(object sender)
 {
       string url = (string)sender;
       //do stuff with url here
  }
0
source

More articles:


All Articles