Problems with huge files
-
I am working on an app that should open a logfile and then it should be possible to: 1. output the whole file as it is 2. reverse the output 3. filter the output I am outputting to a richtextbox since I also want to highlight specific lines. I have tried both using ArrayList and StringBuilder. The latter seems better but then it is not so easy to reverse. To add all text to the RichTextBox I use RTB.Text. I have also thought of outputting all to a temp file and use RTB.LoadFile instead. The problem is that the logfiles can be quite huge, >5MB in extreme cases. This causes it to be very slow and I also encounter out-of-index problems in certain situations that I have not been able to pin down in the debugger. Could anyone advice me on the best way to get a stable and not to slow solution (since I have to work through the whole file line by line to filter it will never be quick)? Code extract: try { StreamReader sr = File.OpenText(fileName); string input = null; //ArrayList myList = new ArrayList(); StringBuilder sb = new StringBuilder(); while ((input = sr.ReadLine()) != null) { if ( Filter or not ) { Filtering of each line. If condition met then sb.Append(input); //myList.Add(input); } } else { sb.Append(input); //myList.Add(input); //myList.TrimToSize(); } } sr.Close(); //myList.TrimToSize(); if ( checkBox4.Checked ) { //myList.Reverse(); } richTextBox1.Clear(); /* for (int i=0;i
-
I am working on an app that should open a logfile and then it should be possible to: 1. output the whole file as it is 2. reverse the output 3. filter the output I am outputting to a richtextbox since I also want to highlight specific lines. I have tried both using ArrayList and StringBuilder. The latter seems better but then it is not so easy to reverse. To add all text to the RichTextBox I use RTB.Text. I have also thought of outputting all to a temp file and use RTB.LoadFile instead. The problem is that the logfiles can be quite huge, >5MB in extreme cases. This causes it to be very slow and I also encounter out-of-index problems in certain situations that I have not been able to pin down in the debugger. Could anyone advice me on the best way to get a stable and not to slow solution (since I have to work through the whole file line by line to filter it will never be quick)? Code extract: try { StreamReader sr = File.OpenText(fileName); string input = null; //ArrayList myList = new ArrayList(); StringBuilder sb = new StringBuilder(); while ((input = sr.ReadLine()) != null) { if ( Filter or not ) { Filtering of each line. If condition met then sb.Append(input); //myList.Add(input); } } else { sb.Append(input); //myList.Add(input); //myList.TrimToSize(); } } sr.Close(); //myList.TrimToSize(); if ( checkBox4.Checked ) { //myList.Reverse(); } richTextBox1.Clear(); /* for (int i=0;i
Obviously, you have to check each line in your file for your filtering condition. You could use XML instead, for which the .NET Framework has built-in support. This is not a valid xml file, but you might understand what I am after: .... .... .... .... .... .... .... Thus, by organizing your data you will be able to find things faster. Additionally, this already is tree-shaped, so you can easily include it into a tree view or something and show error-specific details in a rich-text box. Another way would be to do some organizing yourself. you could have different logfiles for categories, so when the user filters for one category, you simply have to load a file. Also, filtering is much easier if you perform these operations in RAM, not on disk (by loading each line seperatly instead of buffering first, then filtering later). Cheers Sebastian
-
Obviously, you have to check each line in your file for your filtering condition. You could use XML instead, for which the .NET Framework has built-in support. This is not a valid xml file, but you might understand what I am after: .... .... .... .... .... .... .... Thus, by organizing your data you will be able to find things faster. Additionally, this already is tree-shaped, so you can easily include it into a tree view or something and show error-specific details in a rich-text box. Another way would be to do some organizing yourself. you could have different logfiles for categories, so when the user filters for one category, you simply have to load a file. Also, filtering is much easier if you perform these operations in RAM, not on disk (by loading each line seperatly instead of buffering first, then filtering later). Cheers Sebastian
Thanks for your answer but unfortunately I have no control of the logfile. It's because it's huge and diifcult to read I'd like to build my own logfile reader. What I need most I think is a tip on how to reverse output using StringBuilder. Otherwise I do thing StringBuilder is the way to go.