Geek asks and answers

Memory Exception When Using Streams

I have the following algorithm,

private void writetodb()
{
    using(var reader = File.OpenRead("C:\Data.csv");
    using(var parser = new TextFieldParser(reader))
    { 
        //Do some opeartions
        while(!parser.EndOfData)
        {
            //Do operations
            //Take 500 rows of data and put it in dataset
            Thread thread = new thread(() => WriteTodb(tablename, set));
            thread.Start();
            Thread.Sleep(5000);
        }
    }
}

public void WriteTodb(string table, CellSet set)
{
    //WriteToDB
    //Edit: This statement will write to hbase db in hdinsight
    hbase.StoreCells(TableName, set);
}

This method works absolutely normal up to 500 mb of data , but after that it does not speak Out of memory exception.

I am pretty sure that this is due to threads, but using threads is mandatory, and I cannot change the architecture.
Can someone tell me what changes I should make in programming threads in the above program to avoid memory exception.

+4
source share
2 answers

First of all, I can’t understand your words about streaming:

I need to do thread programming in the above program to avoid memory exception.

, TPL, . Thread, . , C# 4.0, TPL . - ( ):

List<Task> tasks  = new List<Task>();
while(!parser.EndOfData)
{
    tasks.Add(Task.Run(() => WriteTodb(tablename, set)));
}
Task.WaitAll(tasks.ToArray());

TPL- TaskScheduler , ThreadPool , .

, , HBase Microsoft, async :

public async Task StoreCellsAsync(string table, CellSet cells)
{
}

, TPL :

List<Task> tasks  = new List<Task>();
while(!parser.EndOfData)
{
    tasks.Add(WriteTodb(tablename, set)));
}
// asynchroniously await all the writes
await Task.WhenAll(tasks.ToArray());

public async Task WriteTodb(string table,CellSet set)
{
    //WriteToDB
    //Edit: This statement will write to hbase db in hdinsight asynchroniously!
    await hbase.StoreCellsAsync(TableName, set);
}

- TPL, :

  • , .
  • , , , .
  • , .
+5

ThreadPool.QueueUserWorkItem. refrence . : https://msdn.microsoft.com/en-us/library/kbf0f1ct(v=vs.110).aspx

0

More articles:


All Articles