Geek asks and answers

Ruby and celluloid

Due to some restrictions, I want to switch my current project from EventMachine / EM-Synchrony to Celluloid, but it's hard for me to contact him. The project I'm coding is a web harvester that should scan tons of pages as quickly as possible.

For a basic understanding of Celluloid, I created 10,000 dummy pages on a local web server and want to crawl them with this simple fragment of Celluloid:

#!/usr/bin/env jruby --1.9

require 'celluloid'
require 'open-uri'

IDS = 1..9999
BASE_URL = "http://192.168.0.20/files"

class Crawler
  include Celluloid
  def read(id)
    url = "#{BASE_URL}/#{id}"
    puts "URL: " + url
    open(url) { |x| x.read }
  end
end

pool = Crawler.pool(size: 100)

IDS.to_a.map do |id|
   pool.future(:read, id)
end

As far as I understand Celluloid, futures are the way to get a response from a dismissed request (comparable to callbacks in EventMachine), right? Another thing is, each actor works in its own thread, so I need some kind of batch request, because 10,000 threads will lead to errors on my OSX dev machine.

, - , ? : URL- 9999, 1300 HTTP- -. - URL-.

+5
1

, , . Celluloid , , # . . , , - :

  crawlers = IDS.to_a.map do |id|
    begin
      pool.future(:read, id)
    rescue DeadActorError, MailboxError
    end
  end

  crawlers.compact.each { |crawler| crawler.value rescue nil }
+7

More articles:


All Articles