upd Now I think that the root of my problem is not "streaming", because I observe a slowdown anywhere in my program. I think that somehow, when using 2 processors, my program runs slower, possibly because the two processors need to "exchange" with each other. I need to do some tests. I will try to turn off one of the processors and see what happens.
======================================
I'm not sure if this is a C # question, maybe this is more hardware related, but I think C # would be the most suitable.
I used a cheap DL120 server, and I decided to upgrade to a much more expensive 2-processor DL360p server. Suddenly, my C # program runs about ~ 2 times slower on a new server, which should be several times faster.
I processed the FAST data of about 60 tools. I created a separate task for each such tool:
BlockingCollection<OrderUpdate> updatesQuery;
if (instrument2OrderUpdates.ContainsKey(instrument))
{
updatesQuery = instrument2OrderUpdates[instrument];
} else
{
updatesQuery = new BlockingCollection<OrderUpdate>();
instrument2OrderUpdates[instrument] = updatesQuery;
ScheduleFastOrdersProcessing(updatesQuery);
}
orderUpdate.Checkpoint("updatesQuery.Add");
updatesQuery.Add(orderUpdate);
}
private void ScheduleFastOrdersProcessing(BlockingCollection<OrderUpdate> updatesQuery)
{
Task.Factory.StartNew(() =>
{
Instrument instrument = null;
OrderBook orderBook = null;
int lastRptSeqNum = -1;
while (!updatesQuery.IsCompleted)
{
OrderUpdate orderUpdate;
try
{
orderUpdate = updatesQuery.Take();
} catch(InvalidOperationException e)
{
Log.Push(LogItemType.Error, e.Message);
continue;
}
orderUpdate.Checkpoint("received from updatesQuery.Take()");
......................
......................
}, TaskCreationOptions.LongRunning);
Since I have about 60 tasks that can be performed in parallel, I expect that 2 * E5-2640 (24 virtual threads, 12 real threads) should run much faster than 1 * E3-1220 (4 real threads). It seems like using DL360p I found 95 threads in task manager. Using the DL120 has only 55 threads.
DL120G7 2 (!!) !
E3-1220 , E5-2640 (3,1 2,5 ), , 2 * E5-2640, , 2 !
HP DL120G7 E3-1220
~ 50 = 24 ~ 80
calling market.UpdateFastOrder = 23 updatesQuery.Add = 25 received from updatesQuery.Take() = 67 in orderbook = 80
calling market.UpdateFastOrder = 30 updatesQuery.Add = 32 received from updatesQuery.Take() = 64 in orderbook = 73
calling market.UpdateFastOrder = 31 updatesQuery.Add = 32 received from updatesQuery.Take() = 195 in orderbook = 204
calling market.UpdateFastOrder = 31 updatesQuery.Add = 32 received from updatesQuery.Take() = 74 in orderbook = 86
calling market.UpdateFastOrder = 18 updatesQuery.Add = 21 received from updatesQuery.Take() = 65 in orderbook = 78
calling market.UpdateFastOrder = 29 updatesQuery.Add = 32 received from updatesQuery.Take() = 76 in orderbook = 88
calling market.UpdateFastOrder = 30 updatesQuery.Add = 32 received from updatesQuery.Take() = 80 in orderbook = 92
calling market.UpdateFastOrder = 20 updatesQuery.Add = 21 received from updatesQuery.Take() = 65 in orderbook = 78
calling market.UpdateFastOrder = 21 updatesQuery.Add = 24 received from updatesQuery.Take() = 68 in orderbook = 81
calling market.UpdateFastOrder = 12 updatesQuery.Add = 13 received from updatesQuery.Take() = 58 in orderbook = 72
calling market.UpdateFastOrder = 22 updatesQuery.Add = 23 received from updatesQuery.Take() = 51 in orderbook = 59
calling market.UpdateFastOrder = 16 updatesQuery.Add = 16 received from updatesQuery.Take() = 20 in orderbook = 24
calling market.UpdateFastOrder = 28 updatesQuery.Add = 31 received from updatesQuery.Take() = 82 in orderbook = 94
calling market.UpdateFastOrder = 18 updatesQuery.Add = 21 received from updatesQuery.Take() = 65 in orderbook = 77
calling market.UpdateFastOrder = 29 updatesQuery.Add = 29 received from updatesQuery.Take() = 259 in orderbook = 264
calling market.UpdateFastOrder = 49 updatesQuery.Add = 52 received from updatesQuery.Take() = 99 in orderbook = 113
calling market.UpdateFastOrder = 22 updatesQuery.Add = 23 received from updatesQuery.Take() = 50 in orderbook = 60
calling market.UpdateFastOrder = 29 updatesQuery.Add = 32 received from updatesQuery.Take() = 76 in orderbook = 88
calling market.UpdateFastOrder = 16 updatesQuery.Add = 19 received from updatesQuery.Take() = 63 in orderbook = 75
calling market.UpdateFastOrder = 27 updatesQuery.Add = 27 received from updatesQuery.Take() = 226 in orderbook = 231
calling market.UpdateFastOrder = 15 updatesQuery.Add = 16 received from updatesQuery.Take() = 35 in orderbook = 42
calling market.UpdateFastOrder = 18 updatesQuery.Add = 21 received from updatesQuery.Take() = 66 in orderbook = 78
HP DL360p G8 2 * E5-2640
~ 95 ; best = 40 ~ 150
calling market.UpdateFastOrder = 62 updatesQuery.Add = 64 received from updatesQuery.Take() = 144 in orderbook = 205
calling market.UpdateFastOrder = 27 updatesQuery.Add = 32 received from updatesQuery.Take() = 101 in orderbook = 154
calling market.UpdateFastOrder = 45 updatesQuery.Add = 50 received from updatesQuery.Take() = 124 in orderbook = 187
calling market.UpdateFastOrder = 46 updatesQuery.Add = 51 received from updatesQuery.Take() = 127 in orderbook = 162
calling market.UpdateFastOrder = 63 updatesQuery.Add = 68 received from updatesQuery.Take() = 137 in orderbook = 174
calling market.UpdateFastOrder = 53 updatesQuery.Add = 55 received from updatesQuery.Take() = 133 in orderbook = 171
calling market.UpdateFastOrder = 44 updatesQuery.Add = 46 received from updatesQuery.Take() = 131 in orderbook = 158
calling market.UpdateFastOrder = 37 updatesQuery.Add = 39 received from updatesQuery.Take() = 102 in orderbook = 140
calling market.UpdateFastOrder = 45 updatesQuery.Add = 50 received from updatesQuery.Take() = 115 in orderbook = 154
calling market.UpdateFastOrder = 50 updatesQuery.Add = 55 received from updatesQuery.Take() = 133 in orderbook = 160
calling market.UpdateFastOrder = 26 updatesQuery.Add = 50 received from updatesQuery.Take() = 99 in orderbook = 111
calling market.UpdateFastOrder = 14 updatesQuery.Add = 30 received from updatesQuery.Take() = 36 in orderbook = 40 <-- best one I can find among thousands
, 2 ? , ~ 60 ? , .NET 95 , 50 24? , 2 1 ? , DL360P Gen8 ?
- market.UpdateFastOrder - orderUpdate
- updatesQuery.Add - orderUpdate BlockingCollection
- updatesQuery.Take() - orderUpdate, BlockingCollection
- orderbook - orderUpdated orderBook