READ TEXT FILE LINE BY LINE AND STORE ALL LINES IN ARRAY...(is a good idea?)

sal21

Ho to reading txt file line by line and store all lines in array. After all line are stored in array, loop in this array line by line for next use... You must know: 1- the dimension of text file is 163.457 MB 2- all line have max 132 lenght string 3 - into file is possible to have a blank line from one line to next line I hope and think is the best way to import this BIG file... I have count the lines are approx 3.364.120 lines!!!!!!!!!!!!!!!

Paul Conrad

Need a little more clarification. What is the requirement for having all of the lines stored in an array? What are you trying to do with all of the lines once they've been read in?

"The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham

sal21

ok... i want to get a part of single line with MID,LEFT,RIGHT ecc and store the piece of line in a var. About array mode, is simple one suggestion.. in Google i have seen with FSO is a good way to speed the reading or not?

Guffa

I, like Paul, still wonder why you need all the lines in an array? Do you really need all the lines in memory at once?

sal21 wrote:

in Google i have seen with FSO is a good way to speed the reading or not?

FSO? Are you using VB6?

Despite everything, the person most likely to be fooling you next is yourself.

Luc Pattyn

Hi,

sal21 wrote:

is a good idea?

No! File.ReadAllLines will read the file and give you the huge array you are looking for; however generating such a huge array will be detrimental to the performance of your app, no matter how you do it. If a single pass through the file is all you need, have a look at streams. If you only need a fraction of the file's content, read it line by line, don't store all the lines, simply store the pieces you need (use StreamReader.ReadLine) :)

Luc Pattyn [Forum Guidelines] [My Articles]

Fixturized forever. :confused:

Paul Conrad

Guffa wrote:

Do you really need all the lines in memory at once?

I am still waiting for a real answer to that question I asked him. His requirements still make little sense other than picking off pieces of the text, but why everything all at once?

"The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham

Jon_Boy

sal21 wrote:

I hope and think is the best way to import this BIG file...

When you say "import", what *exactly* are you implying. Are we talking just getting the data from the file into your app to use variables/collections of some kind. Then I would agree with the above, that using a streamreader and reading line by line would be the best bet. OR, if your goal is to get this data into a DB like SQL? If this is the case, I would scrap what you're doing an look into DTS/SSIS since both give you programmatic control of the data coming in and format/manipulate as needed. Speed wise, there's not even a comparison.

Any suggestions, ideas, or 'constructive criticism' are always welcome. "There's no such thing as a stupid question, only stupid people." - Mr. Garrison

Jon_Boy

Paul Conrad wrote:

I am still waiting for a real answer to that question I asked him.

To benchmark how fast the memory can be filled?

Any suggestions, ideas, or 'constructive criticism' are always welcome. "There's no such thing as a stupid question, only stupid people." - Mr. Garrison

Paul Conrad

:laugh: More along the lines of what is the purpose of reading the entire file into memory at once.

"The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham