Geek asks and answers

Performance of mass loading data from XML file in MySQL

Should importing 80 megabyte XML data into MySQL take more than 5 days?

I am currently importing an XML file of approximately 80 GB in size, the code I use is in this gist and while everything is working properly, it works for almost 5 days in a row and it doesn’t even get close to execution ...

The average table size is approximately:

Data size: 4.5GB
Index size: 3.2GB
Avg. Row Length: 245
Number Rows: 20,000,000

Let me know if more information is needed!

Server Features:

Please note that this is a linear VPS

Intel Xeon Processor L5520 - Quad Core - 2.27 GHz 4 GB Total Ram

XML example

https://gist.github.com/2510267

Thank!

, , .

+3
3

, , - , . .

, , , - ; , . ON DUPLICATE KEY UPDATE ( MySQL, ), INSERT .

, XML , mysqlimport, . , XML, , , , ( INSERT UPDATE, mysqlimport) LOAD DATA INFILE).

+2

(), , , iterparse . , , node ( , , ), .

0

3 , , . -

Performing these tasks greatly improved the speed of a similar project I was working on. Perhaps if you posted the xml code and example, I could offer a deeper solution. (edit, sorry, missed the point ...)

0
source

More articles:


All Articles