Problems with CSV file blowing out memory

[LongRange Shooter]

I have two .csv files .. one is a filter content and the other is national data. The national data file crashes when Excel tries to open it BUT if my PM "uses Access" he can open all of the data. I used a StreamReader and did an inline ReadLine() but that got the same result as Excel -- out of memory error. Then I read that if you get an out of memory failure using ReadLine that using the more robust Read() will resolve the issue. I changed my code to use an inline method called GetLine() so that I didn't change my code too drasitcally and ran a test. It still runs out of memory. (code is below) Anyone have any ideas on how I can get past the I/O failure?

    private string GetLine(StreamReader reader)
    {
        if (reader.Peek() < 0) return null;
        
        List<char> buffer = new List<char>(limit);

        while (reader.Peek() >= 0)
        {
            buffer.Add((char)reader.Read());
            if (reader.Peek() < 0) return Debuff(buffer);

            if (buffer.Count > 3 && 
                (buffer\[buffer.Count - 1\].Equals('\\r') && buffer\[buffer.Count - 2\].Equals('\\n')) || 
                (buffer\[buffer.Count - 1\].Equals('\\n') && buffer\[buffer.Count - 2\].Equals('\\r')))
                return Debuff(buffer);
        }
        return Debuff(buffer);
    }

[Ed Poore]

First question how big does buffer get?

---

My Blog[^]

[Skippums]

The advice that you received that the Read method is more robust is due to the following: The readline method goes until it finds a newline character. If you have a 2,000 GB file with no newline characters, then ReadLine attempts to read it ALL into memory (which is clearly going to throw an out of mem exception). The code you provided above is using Read, this is true. But you are still getting the same exception because you basically just reimplemented the same functionality, and are still trying to load a 2000 GB file into memory. The reason why Read is suposedly "safer" is because typically, you only need a portion of the file at any given time. Read into memory only that which you need, or at minimum put a limit to the file size you can handle. Hope this helps,

Sounds like somebody's got a case of the Mondays -Jeff

[LongRange Shooter]

Well, believe it or not your question solved my problem. I added a try{}catch{} and found that the buffer exceeded 256MEG in size. Much larger than any record in the file could possibly be. So I traced the characters and found that it only contained a \n by itself at the end of each line. Found many other inconsistancies in the file (like quoted numbers) but the application is now processing the file without exceeding memory! Thanks for the help. Michael