Geek asks and answers

Processing and combining two large files

I need to read in two large files (more than 125 MB). Each file contains records that have similar data. I need to find the records that are in both of them, and then if the record fields do not match, I need to overwrite the records in the file with two fields contained in the records from the file.

For example, the first file has the following fields:

ID, ACCT, Bal, Int, Rate

The second file has the following fields:

TYPE, ID, ACCT, Bal, Int, Rate.

So, if the entry in file 1 has the same ACCT number as the entry in file 2, then Bal, Int and Rate in file 2 should be overwritten with the value Bal, Int and Rate from file 1.

Some entries will not be in every file. The output file I need to create is all two records from the file, and if the record is not in the file, then it will be written to the file as is, but then the records that need to be changed will be included.

I tried many different options, but most of them are not efficient enough to work with large files. What is the right direction to solve this problem? Thanks in advance for any help.

+3
source share
2 answers

Download all entries from file 1 to the hash table with the ACCT key.
Go through all the entries in file 2 and update if necessary.

Difficulty: O (n)

NTN

0
source

Define two types of specific classes: one for each file.

class FileOne
{ 
    public int LineNumber {get;set};
    public int Id{get;set;}; 
    public double Bal {get;set;};
...
}

class FileTwo
{ 
    public int LineNumber {get;set};
    public string TranType{get;set;};  // type = reserved word
    public int Id{get;set;}; 
    public double Bal {get;set;};
...
}

IList < > , IList myFileOne IList myFileTwo, , , .

linq :

var diffs = from f1 in myFileOne
            join f2 in myFileTwo on f1.Id = f2.Id
            where f1.Bal != f2.Bal // add whatever conditions you need here
            select new {
                f1.Id, f2.Bal, f2.Int, f2.Rate, f1.LineNum
            }

Diffs 4 select. f1.LineNum myFileOne , f2.

, ?

+1

More articles:


All Articles