Geek asks and answers

Memory Leak (?) With StreamReader

I have some very large files of size 500MB++containing integer values ​​(actually this is a bit more complicated), I read these files in a loop and calculate the maximum value for all files. For some reason, memory grows constantly during processing, it seems that GC never releases memory received by previous instances lines.

I can not transfer data and use GetFileLinesfor each file. If the actual amount of memory required for storage linesfor a single file is 500MB, why am I getting 5GBfrom RAM, which is used after processing 10 files? In the end, it crashes out of memory after 15 files.

Payment:

   int max = int.MinValue;

   for (int i = 0; i < 10; i++)
   {
      IEnumerable<string> lines = Db.GetFileLines(i);

      max = Math.Max(max, lines.Max(t=>int.Parse(t)));
   }

GetFileLines Code:

   public static List<string> GetFileLines(int i)
   {
      string path = GetPath(i);

      //
      List<string> lines = new List<string>();
      string line;

      using (StreamReader reader = File.OpenText(path))
      {
         while ((line = reader.ReadLine()) != null)
         {
            lines.Add(line);
         }

         reader.Close();
         reader.Dispose(); // should I bother?
      }

      return lines;
   }
+5
6

ReadLines , , :

  Math.Max(max, File.ReadLines(path).Max(line => int.Parse(line)));

:

http://msdn.microsoft.com/en-us/library/dd383503.aspx

Edit:

ReadLines :

    public static IEnumerable<string> ReadLines(string fileName)
    {
        string line;
        using (var reader = File.OpenText(fileName))
        {
            while ((line = reader.ReadLine()) != null)
                yield return line;
        }
    }

,

+5

, , (, , , , ?). , StreamReader.

, ? , IEnumerable<string> List<string> . , , , .

, Close Dispose ; using .

+4

, :

int max = Int32.MinValue;
using(var reader = File.OpenText(path)) 
{
    while ((line = reader.ReadLine()) != null)
    {
         int current;
         if (Int32.TryParse(line, out current))
             max = Math.Max(max, current);
     }    
}
+1

memmory ( )

, ?

.

0

, . () .

, .

, , , - , , , , .

0
source

Well, if you need a solution in which you can immediately read the entire file, because you are sure that you need this performance increase, then let it be done so that you do not have a memory problem.

public static int GetMaxForFile(int i) 
{ 
    string path = GetPath(i); 

    var lines = new List<string>(File.ReadAllLines(path));

    // you MUST perform all of your processing here ... you have to let go
    // of the List<string> variable ...
    int max = Math.Max(max, lines.Max(t=>int.Parse(t)));

    // this may be redundant, but it will cause GC to clean up immediately
    lines.Clear();
    lines = null;

    return max;
}
0
source

More articles: