Geek asks and answers

Loading large datasets into a Rails application

I am dealing with millions of rows of data that I want to load into my Rails application as models. I use MySQL as a database, and I'm on Rails 2.3.14.

One of my employees says that it is not practical to add records directly to MySQL, bypassing the Rails ActiveRecord system. He briefly talks about the specifics, but the bottom line is that Rails does a lot of “magic” when you use the ActiveRecord system, and that confuses Rails if you enter data outside of this system. Can anyone clarify how true this is?

If I have to load data into Rails via ActiveRecord, I read that the activerecord-import plugin is the way for this type of work.

Any feedback on the best approach to loading massive amounts of data in Rails would be welcome.

+3
source share
3 answers

I can think of six main points to consider, the last five relate to the magic of rails:

  • Speed . This is huge. Active recording, one-on-one inserts may take seconds for each row. So a million seconds for a million lines is 11.5 DAYS , which would give him a bad rap for many people!

  • Validation . You need to make sure that the database performs the same checks as in your models / existing data.

  • Timestamps. , created_at/updated_at ,

  • Counter Caches. .

  • ActiveRecord , act_as_audited, Model, , ActiveRecord.

  • - . ( ), . , , , .. , ActiveRecord .

+3

, MySQL , , ActiveRecord . MySQL, , , ActiveRecord. , script , .

. students teachers. students teachers.students_count ActiveRecord .

, , . - FUD.

ActiveRecord - ActiveRecord, . , / script rake arel ( ActiveRecord) , activerecord-import. , (, , ) , , .., .

+1

, . .

. , . rails, , . , , .

. , - ( ​​db), .

, , , , - "" , - , , .

0

More articles:


All Articles