Geek asks and answers

Out of memory error while reading a very large text file in vb.net

I was instructed to process a text file with a fixed width of 3.2 GB. Each line has a length of 1563 characters, and in a text file about 2.1 million lines. After reading about 1 million lines, my program crashes due to a memory exception error.

Imports System.IO
Imports Microsoft.VisualBasic.FileIO

Module TestFileCount
    ''' <summary>
    ''' Gets the total number of lines in a text file by reading a line at a time
    ''' </summary>
    ''' <remarks>Crashes when count reaches 1018890</remarks>
    Sub Main()
        Dim inputfile As String = "C:\Split\BIGFILE.txt"
        Dim count As Int32 = 0
        Dim lineoftext As String = ""

        If File.Exists(inputfile) Then
            Dim _read As New StreamReader(inputfile)
            Try
                While (_read.Peek <> -1)
                    lineoftext = _read.ReadLine()
                    count += 1
                End While

                Console.WriteLine("Total Lines in " & inputfile & ": " & count)
            Catch ex As Exception
                Console.WriteLine(ex.Message)
            Finally
                _read.Close()
            End Try
        End If
    End Sub
End Module

This is a fairly simple program that reads a text file one line at a time, so I assume that it should not take up too much memory in the buffer.

In my life, I cannot understand why it is crumbling. Does anyone have any ideas?

+5
source share
2 answers

, , peek, : ( #, VB)

while (_read.ReadLine() != null)
{
    count += 1
}

, ,

while ((lineoftext = _read.ReadLine()) != null)
{
    count += 1
    //Do something with lineoftext
}

, 1563 ( ), ASCII ( ), ( #, )

long bytesPerLine = 1563;
string inputfile = @"C:\Split\BIGFILE.txt"; //The @ symbol is so we don't have to escape the `\`
long length;

using(FileStream stream = File.Open(inputFile, FileMode.Open)) //This is the C# equivilant of the try/finally to close the stream when done.
{
    length = stream.Length;
}

Console.WriteLine("Total Lines in {0}: {1}", inputfile, (length / bytesPerLine ));
+1

ReadAsync, DiscardBufferedData ( )

Dim inputfile As String = "C:\Example\existingfile.txt" 
    Dim result() As String 
    Dim builder As StringBuilder = New StringBuilder()

    Try
        Using reader As StreamReader = File.OpenText(inputfile)
            ReDim result(reader.BaseStream.Length)
            Await reader.ReadAsync(result, 0, reader.BaseStream.Length)
        End Using 

        For Each str As String In result
            builder.Append(str)         
        Next
      Dim count as Integer=builder.Count()
       Console.WriteLine("Total Lines in " & inputfile & ": " & count)
    Catch ex As Exception
            Console.WriteLine(ex.Message)
    End Try
0

More articles:


All Articles