Why does recque use child processes to process each job in the queue?

We used Resque in most of our projects, and we were pleased with it.

In a recent project, we had a situation where we connect to the direct streaming API from Twitter. Since we must maintain the connection, we dropped every line from the streaming API to the resque queue so that the connection was not lost. And we were processing the queue afterwards.

We had a situation where the speed of entry into the queue was about 30-40 / sec, and the speed with which the queue was knocked out was only 3-5 / second. And because of this, the line has always increased. When we checked the reasons for this, we found that resque has a parent process, and for each queue job, it creates a child process, and the child process will process the job. The environment of the rails was quite difficult, and the child process took time.

So, we have completed another task of rake of this kind, so far:

rake :process_queue => :environment do
  while true
    begin
      interaction = Resque.pop("process_twitter_resque")
      if interaction
        ProcessTwitterResque.perform(interaction)
      end
    rescue => e
      puts e.message
      puts e.backtrace.join("\n")
    end
  end
end

and started this task as follows:

nohup bundle exec rake process_queue --trace >> log/workers/process_queue/worker.log 2>&1 &

This does not handle failed jobs and that’s it.

, , Resque . ( , , , , , Resque ).

, Resque . / ?