Geek asks and answers

What is the proper way to test a system with a simultaneous workflow?

I am working on a system consisting of:

  • Database
  • A bunch of threads pumping rows to this database
  • A bunch of threads working on these lines

The most important point is that no more than one employee must work simultaneously in a row.

Of course, there are many strategies here ... I can block things at the database level, I can use mutexes, etc.

But no matter how I implement it, I need to be able to test the system to make sure that I did it right.

What is the correct way to test this?

Everything I did is performed by hundreds of threads and constantly checks for random overlap. The problem is that she hit or missed. This is a probability. Even if I start 500 threads in an hour, there may still be a thread that overlaps another, only rarely.

Also, how can I check for overlap correctly? The "overlap controller" has a final resolution that may not be present in the actual overlaps that occur ...

I know that concurrency is a complex topic, but, of course, there should be some recommendations or recommended ways to test such a system, except that it starts up and crosses fingers for a long time ...

+5
source share
10 answers

, : /ACID (). - ? " " ?

+4

?

. , , - - , , . , , RPC .

Java .

 try {
     // add it to some atomic concurrent hash-map
     if (workingRowIdMap.putIfAbsent(rowId, null) != null) {
         // scream and shout and log a concurrency failure
     }
     // do the _real_ database stuff here
 } finally {
     workingRowIdMap.remove(rowId);
 }

, SQL . , SQL, , . , .

, , , , .

- , , - . , MySQL , . , , - .

, ? " " , , ...

. - - . , . , , . .

, concurrency . , try/finally .., .

0

, , - , , .

. monte-carlo, .

, , . : WorkerCount int not null. . , ( - ). , WorkerCount IN (0, 1). , .

.

0

, , , , , #

SQLBulkCopy (ADO.Net) (SQL Server).

.

, ConcurrentList Lock (object) , 5 , SQLBulkCopy.

, ...

0

, , .

/. , ( ), concurrency. , .

, , . , . , , .

, PROCESSING_STATUS PROCESSING_STATUS = 0. , . ( / ) , , PROCESSING_STATUS = 0. ​​ . PROCESSING_STATUS 1. , , .

, , , . , . Python, # Java. .

, , , , PROCESSING_STATUS, , 2. , , . , , .

, , , PROCESSING_STATUS. PROCESSING_STATUS = 0 - , 1, , / . 2 , . , .

, , , . , , .

, Queues . , , = = .

, , PROCESSING_STATUS. , () PROCESSING_STATUS = 0. , 0, . - :

update TABLE_X set PROCESSING_STATUS = MY_UNIQUE_THREAD_ID
where key in (select key from TABLE_X where PROCESSING_STATUS = 0 LIMIT 5)
      and PROCESSING_STATUS = 0

, . , PROCESSING_STATUS = MY_UNIQUE_THREAD_ID. . , PROCESSING_STATUS MY_UNIQUE_THREAD_ID. , concurrency , , . , : .

( ). (k% n_readers) select. :

SELECT * from TABLE_X WHERE (key % N) == MY_UNIQUE_THREAD_ID
0

,

  • , , , (, )
  • , (, )

, , . ...

, , , ConcurrentDictionary , - . ConcurrentQueues, /.

0

row_id, .

.NET - :

private var rowWorkers = new Dictionary<int,Task>();

public void ScheduleWorkOnRow(int id) 
{
  // starting empty worker to be able to continue on it
  if(rowWorkers[id] == null) rowWorkers.Add(id, Task.Run(() => { });
  // scheduling continuation
  rowWorkers[id].ContinueWith(WorkOnRow, id);
}

private void WorkOnRow(Task task, object id)
{
  //your code
}

, , .

0

, , - , . , . :

:

, .

, , , . , , .

, . . ThreadCheck, , , .

1 -

, , ​​ , ThreadCheck Thread [Thread ID]. , ThreadCheck . , , "". , , .

2 -

, , - . FIFO ( , ), . . , (). , , , .

, + 1, . 1, 2, 3 X.

, , - , , "!!!". , . , , .

++ - , . , , 1234567890 , . , .

3 -

. , . . , , , , .

, , - , .

, , - , - .

0

, Heisenbug, , Chess. , , - , , .

, CHESS Microsoft .

, win32, . NET. codeproject , ,

0

. , 2 . 1) ? 2) , , ?

, , - , , . , , . .

:

, . , . , . , . , 100% .

, , . , , . , .

:

Create datasets with known lines of overlapping data, and then try something like the Grimace Chess tool. But you need the data that you KNOW, will cause problems to see how the code processes them and see if they process them correctly. Don't just keep throwing random data at him in the hope that something will stick. For example, what happens if you run 500 threads that are all trying to access the same line, perhaps because you have the same primary key?

0
source

More articles:


All Articles