Geek asks and answers

How to handle large page impressions in memcache before merging in mysql

I am writing an analytics tool that will keep a log every time a page is hit. The problem I'm experiencing is the amount of impressions that can get very high, and I may have to process millions per day, or at least ~ 300 seconds (I have people who say they can reach 10 thousand per second). My current setup is to register php every time the page is accessed directly into the innodb MySQL table, it becomes slow, and the server seems to overshoot sometimes, which is not so bad for analytics, but not perfect. My idea is to bring memcache to the forefront and record all the impressions first, and then start the batch process, which takes memcache impressions and then puts them in db in a quiet period, for example.3 a.m. So my question is:

1) Is this correct? Ideally, I do not want to introduce any new technologies or implement NoSQL dlls such as mongo or couch.

2) How would this be done in memcache? Since there is no way to find out all the keys that are in the cache, I thought of setting a counter that increases by each impression, and then a key with a counter as an identifier containing data, the batch process would loop at the last position to the counter and process these records, and then set your last position to the current counter.

Any help is appreciated.

Greetings

+3
source share
1 answer

I would not create a serperate series, but instead integrated it into your code.

Please note that there is no check / check / block. Some pseudo codes:

$mViews = $oMemcache->get('views');
if ($mViews > 1000) {
    $oDatabase->update($mViews);
    $oMemcached->replace('views', 1);
}
else {
    $oMemcached->increment('views');
}
0
source

More articles:


All Articles