Geek asks and answers

How to parse a large XML file and save it in a database

I have a rather large xml file (more than 2 MB) that I process and save in the sqlite database. I can take it apart and save it for the first time. My question is about updating the database when I want to parse the xml file again (for changes, additions or deletions). My initial thought was to simply erase the information in the database and reinsert it, rather than analyze the data, check whether the item is already in the database and perform an update. Is there an approach that is better than the other? Will performance be affected in one way or another? I would appreciate any thoughts on this.

+3
source share
2 answers

Yes, reinserting is probably a bad idea. How complex is the xml structure, how many tables are involved when you ask for the existence of one element, which is reflected in the structure?

If this is complicated, perhaps you can create a checksum of your records or a hash of some attributes and values ​​that uniquely identify the record and store this hash / checksum in an additional table in db, when you look for modified records you just calculate the hash / checksum and look for her in one table. Perhaps this even makes the request faster, depending on how expensive the hash calculation is.

+2
source

Inserting only what needs to be changed will obviously be faster than dropping the entire database and inserting it again. At least my thoughts.

, , , , . , .

+2

More articles:


All Articles