Large TXT files
-
Hello All i am trying to read a large txt files -may 1 GB-, which coze my program to hang, and i need to know if there are techniques that make this without hang and do it faster if you know something can help. please tell me.
The best thing is to avoid such files. You *can* read a file a line at a time, but I'm not sure if it loads in to memory anyhow when you do that. I'd hope not. What code are you using right now ? What is in the files ?
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
-
Hello All i am trying to read a large txt files -may 1 GB-, which coze my program to hang, and i need to know if there are techniques that make this without hang and do it faster if you know something can help. please tell me.
Hi You could consider using a backgroundworker to read the file, while showing a "loading" sign or something.. :-D basically something like: private void getTextFileWorker_DoWork(object sender, DoWorkEventArgs e) { //start to read the file, show loading sign... while(morelinesinfile) { this.Invoke(new AddValueDelegate(AddValue), new object[] { valueFromFile }); } } private void AddValue(string valueFromFile) { //do something with the value, add it to a list... } private void getTextFileWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e) { //finish up etc..hide sign.. } best regards HV
-
Hi You could consider using a backgroundworker to read the file, while showing a "loading" sign or something.. :-D basically something like: private void getTextFileWorker_DoWork(object sender, DoWorkEventArgs e) { //start to read the file, show loading sign... while(morelinesinfile) { this.Invoke(new AddValueDelegate(AddValue), new object[] { valueFromFile }); } } private void AddValue(string valueFromFile) { //do something with the value, add it to a list... } private void getTextFileWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e) { //finish up etc..hide sign.. } best regards HV
I suspect the real issue is that 1 GB of text will, on most computers, mean your RAM is full and you're using virtual memory.
Christian Graus - Microsoft MVP - C++ Metal Musings - Rex and my new metal blog "I am working on a project that will convert a FORTRAN code to corresponding C++ code.I am not aware of FORTRAN syntax" ( spotted in the C++/CLI forum )
-
Hi You could consider using a backgroundworker to read the file, while showing a "loading" sign or something.. :-D basically something like: private void getTextFileWorker_DoWork(object sender, DoWorkEventArgs e) { //start to read the file, show loading sign... while(morelinesinfile) { this.Invoke(new AddValueDelegate(AddValue), new object[] { valueFromFile }); } } private void AddValue(string valueFromFile) { //do something with the value, add it to a list... } private void getTextFileWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e) { //finish up etc..hide sign.. } best regards HV
that is my code , and i had run it for a txt file its size is 423 MB, and i left my PC opened and in the mext day i found that it records about 10 hours and large number of lines about 500000 line and also i got an exception "OutOfMemoryException" and i dont know much about "backgroundworker", if you know a complete example plz provid me by the link StreamReader SR; private void btnReadFile_Click(object sender, EventArgs e) { DateTime DT= DateTime.Now; TimeSpan T = new TimeSpan(DT.Day, DT.Hour, DT.Minute, DT.Second); if (SR != null) { while (!SR.EndOfStream) { try { string line = SR.ReadLine(); txtFileContent.AppendText(line); txtFileContent.AppendText("\n"); txtFileContent.AppendText("\r"); int count = Convert.ToInt32(labNumOfLines.Text); count++; labNumOfLines.Text = count.ToString(); DateTime tempTime = DateTime.Now; TimeSpan T2 = new TimeSpan(tempTime.Day, tempTime.Hour, tempTime.Minute, tempTime.Second); T2=T2.Subtract(T); labTime.Text = T2.ToString(); this.Refresh(); } catch(Exception ex) { MessageBox.Show(ex.Message); } } } }
-
Hello All i am trying to read a large txt files -may 1 GB-, which coze my program to hang, and i need to know if there are techniques that make this without hang and do it faster if you know something can help. please tell me.
Also if the file is order or can be ordered access it in that manner. Ie, Binary Searches, index based searches, etc. I have worked with several multi gig files in the past and that is the only hope.
File Not Found
-
that is my code , and i had run it for a txt file its size is 423 MB, and i left my PC opened and in the mext day i found that it records about 10 hours and large number of lines about 500000 line and also i got an exception "OutOfMemoryException" and i dont know much about "backgroundworker", if you know a complete example plz provid me by the link StreamReader SR; private void btnReadFile_Click(object sender, EventArgs e) { DateTime DT= DateTime.Now; TimeSpan T = new TimeSpan(DT.Day, DT.Hour, DT.Minute, DT.Second); if (SR != null) { while (!SR.EndOfStream) { try { string line = SR.ReadLine(); txtFileContent.AppendText(line); txtFileContent.AppendText("\n"); txtFileContent.AppendText("\r"); int count = Convert.ToInt32(labNumOfLines.Text); count++; labNumOfLines.Text = count.ToString(); DateTime tempTime = DateTime.Now; TimeSpan T2 = new TimeSpan(tempTime.Day, tempTime.Hour, tempTime.Minute, tempTime.Second); T2=T2.Subtract(T); labTime.Text = T2.ToString(); this.Refresh(); } catch(Exception ex) { MessageBox.Show(ex.Message); } } } }
Is
txtFileContentastringor aStringBuilder? IF the former replacing it with the latter will see a massive performance boost. A string is immutable, which means each time you dotxtFileContent.AppendText(line);you're creating a new string and copying the old one to it. For normal use immutable strings allow certain optimizations that result in faster code, but when building a really large string they absolutely murder performance. IF you;re using a string for each line of your file you're copying everything read so far three times. And absolutely thrashing the GC in the process.-- CleaKO The sad part about this instance is that none of the users ever said anything [about the problem]. Pete O`Hanlon Doesn't that just tell you everything you need to know about users?
-
Is
txtFileContentastringor aStringBuilder? IF the former replacing it with the latter will see a massive performance boost. A string is immutable, which means each time you dotxtFileContent.AppendText(line);you're creating a new string and copying the old one to it. For normal use immutable strings allow certain optimizations that result in faster code, but when building a really large string they absolutely murder performance. IF you;re using a string for each line of your file you're copying everything read so far three times. And absolutely thrashing the GC in the process.-- CleaKO The sad part about this instance is that none of the users ever said anything [about the problem]. Pete O`Hanlon Doesn't that just tell you everything you need to know about users?
-
That almost certainly uses a string internally. Load the entire thing into a stringbuilder first and then copy when complete. Even then I seriously doubt the tb is capable of handling that large a string in a managable fashion.
-- CleaKO The sad part about this instance is that none of the users ever said anything [about the problem]. Pete O`Hanlon Doesn't that just tell you everything you need to know about users?