Geek asks and answers

Celery A task that communicates with Twitter

What is the right approach when writing celery tasks that communicate with the service, which have speed limits and sometimes are absent (do not respond) for a long period of time?

Do I need to use task replay? What if the service skips too much time? Is there a way to save these tasks for later execution after a long period of time?

What if this is a subtask in a long task?

+3
source share
1 answer

First, I suggest you set the socket timeout to avoid a long wait for a response. Than you can catch a TimeOutException socket and in this case to retry with a lot of time, for example, 15 minutes. In any case, I usually use incrementalRetry with a percentage of the increase, this will increase the time each time the task is repeated, this is useful when you write a task that depends on external services that may be available for a long time. You can set a large number of attempts for the task, for example 50, and set the default retry time using var

#20 seconds
self.default_retry_delay = 20

after you can implement such a method for your task

def incrementalRetry(self, exc, perc = 20, args = None):
    """By default the retry delay is increased by 20 percent"""
    if args:
        self.request.args = args

    delay = self.default_retry_delay

    if self.request.kwargs.has_key('retry_deleay'):
        delay = self.request.kwargs['retry_deleay']

    retry_delay = delay+round((delay*perc)/100,2)
    #print "delay"+str(retry_delay)

    self.retry(self.request.args,
               self.request.kwargs.update({'retry_deleay':retry_delay}),
               exc=exc,countdown=retry_delay, max_retries=self.max_retries)

What if this is a subtask in a long task?

, task.delay(args = []) , , , , - .

+2

More articles:


All Articles