Geek asks and answers

SQLAlchemy + asks for an asynchronous template

I am currently working on an application in which a client makes some kind of call to web services, some little processing is done on the returned JSON data, and then stored in a database. I am currently using SQLAlchemy queries. The amount of processing is very small (just changing the data to a more relational format). I do not use ORM for SA. I just use engine + transactions.

I was interested that a good template for this would be asynchronous (the request is returned → transferred to the database → the next request is launched, without waiting for the database transaction to complete).

I know that in Python there are many tools (multiprocessing, threads, coroutines, asyncore, etc.). However, it’s hard for me to find a good tutorial for my use case.

I was wondering if anyone has any suggestions, libraries that I should pay attention to, or asynchronous templates that would help me solve this problem.

Thank.

+1
source share
1 answer

You can click each request in Queue and provide a set of workers who process each of them and push them into the database.

Here is a simple example of a working fluid:

import threading
import time
from Queue import Queue, Empty
from random import choice

class worker(threading.Thread):
  def __init__(self):
    threading.Thread.__init__(self)
    self.q = Queue()
  def run(self):
    while True:
      try:
        r = self.q.get_nowait()
      except Empty:
        r = None
      if r is None:
        time.sleep(1.0)
        continue
      # do something with 'r'
      print '%s: Handled  request %s' % (self, r)
  def push(self, r):
    self.q.put(r)

workers = [worker() for i in range(5)]
for w in workers:
  w.start()

Then distribute the requests for these workers as follows:

choice(workers).push(req)
+1
source

More articles:


All Articles