I have an application used to import a large array of data (millions of records) from one database to another, which makes diff in this process (IE deletes everything that has been deleted, updates things, etc.). Due to many foreign key restrictions and the like, and to speed up application processing, it loads the entire destination database into memory, and then tries to load portions of the original database and performs a comparison in memory, updates the destination in memory. At the end, he writes these changes back to the destination. Databases do not correspond to each other, therefore one table in one can be several tables in another, etc.
So, to my question: it takes several hours to complete this process (sometimes close to the day depending on the amount of added / changed data), and this makes debugging very difficult. Historically, when we encounter an error, we made changes, and then re-launched the application, which should load all the data into memory again (taking quite a lot of time), and then start the import process until we get to that part, and then we cross our fingers and hope that our changes work. This is not fun :(
To speed up the debugging process, I make architectural changes by moving the import code to a separate dll, loaded into a separate appdomain so that we can unload it, make changes and reload it and try to start the import section again, going up to where we left off and we'll see will we get the best results. I thought I was a genius when I came up with this plan :) But he has a problem. I either have to load all the data from the target database into the second appdomain, and then, before unloading, copy it all to the first using the [Serializable] deal (this is really very slow when unloading and reloading the dll) or loading the data in the appdomain host and refer to it in the second using MarshalByRefObject (which, as it turned out, made the whole process look slow)
, : ? , ! , .
, , , , . , ( , ), , , MarshalByRefObject. - , .
- , dll ++ dll . , IE, dll, - , , , , dll, ?
, . , , , . .