I am building a pipeline using the TPL data stream. Let's say I have a pipeline consisting of four TransformBlocks
GetDataFromDatabase
ProcessData1
ProcessData2
SendDataToRestService
Each of them can (probably will) be a long work. GetDataFromDatabase is probably mostly waiting for a database response, SendDataToRestService is waiting for a service response. ProcessData 1 and 2 will be long-running local processor intensive processes.
Now I decided to set maxdegreeofparalellism on each of my blocks to the number of cores on the machine (which is likely to be 4).
But it occurs to me that this means that he will do 4 GetDataFromDatabases and 4 ProcessData1 and 4 ProcessData2 and 4 SendDataToRestService at once. This would mean that I could have 16 threads running at the same time. Add to this that I actually have 3 different threads that do similar things at the same time.
QUESTION: What is the best approach to this, so that I get the best performance? From what I read, if I try to do too much, I can actually slow down performance due to context switching. Is there a better approach, or is it "suck it and see"?
source
share