Geek asks and answers

Email Queue - X Workers and Overlaps

I have a large queue of emails loaded into a MySQL table, since events are happening around my site. At any time, perhaps 1k-2k is waiting for dispatch at any time.

I worked on batch sets of emails and sent them through one employee. Now I want to add X workers to the system with an increase in load.

My question is to prevent employees from picking the same email address. I heard the following options.

  • Use SELECT FOR UPDATE to lock a row
  • Use a timestamp to inform employees that this was done and when
  • Have a parent who tracks children and delegates work.

I see the value in each of them, but also a lot of read / write in db. So what would you do?

Doing this through PHP and MySQL

+3
source share
1 answer

Another quick and dirty option, assuming that you have a primary key, which is either random or structured, allows each of the N workers to collect only letters where key%N==ithe iidentifier of the worker is.

You can, of course, take another hash value, which is quite random. No need to block, no extra coordination, just a tiny problem when one worker dies and you don’t notice ...

+2
source

More articles:


All Articles