Geek asks and answers

Ruby-on-Rails 3.2: CSV export with a large dataset (100,000 records)

Introduction

I have an application that has several tables, some with some and some without associations.

Some tables should contain about 100,000 records.

The application uses Rails 3.2 on Ruby 1.9 and is hosted on Heroku. I have access to employees, if necessary.

Requirement in question

The vital requirement for the application is to enable users to export data as CSV - this requirement allows the user to filter the data that they want to export, but I'm not worried about it at the moment, since you will see from the data below, I programmed hard, what data needs to be exported, but this eliminates the possibility of creating a rake task to simply export the entire table.

Also, the implemented method must be taken into account in order to allow the use of several tables in order to avoid the repetition of unnessaccery code.

Current solution

delayed_job CSV . , , , http://www.ayokasystems.com/blog/delegating-long-running-jobs-in-rails/ 'abdullah'.

, CSV LONGTEXT UserJobs, .

, , , 100 000 . , find_each , , :

[Worker(host:*** pid:18637)] ReportJob failed with NoMethodError: undefined method `each' for #<Title:0x007ff20c1ec1b0> - 0 failed attempts
[Worker(host:*** pid:18637)] ReportJob failed with NoMethodError: undefined method `each' for #<Title:0x007ff20ec47f18> - 1 failed attempts
[Worker(host:*** pid:18637)] 2 jobs processed at 10.5219 j/s, 2 failed ...

:

def perform
  Title.find_each do |titles|
    csv_data = CSV.generate do |csv|
      titles.each do |t|
        csv << t.to_csv
      end
    end
    user_job = UserJob.find(user_job_id)
    user_job.update_attribute :data, csv_data
  end
end

- , , , , .

, , , , , Heroku.

+3
2

, title (not array).

csv_vals = []
columns = [:name, :release_date, :studio]

Title.find_each(:select => columns) do |title| 
  columns.each {|value| csv_vals << "#{title[value]}"}
end

# comma separated string 
csv_string = csv_vals.join(',')

CSV, , .

, SELECT , . 100 000 , DB DB. find_each , .

+3

find_each , , each . find_in_batches, , :

Title.find_each do |title|
  CSV.generate do |csv|
    csv << title.to_csv
  end
  user_job = UserJob.find(user_job_id)
  user_job.update_attribute :data, csv_data
end
+1

More articles:


All Articles