Dealing with Large Amounts of Data in Files
-
I am currently debugging a program that stores large amounts of data. Every 15 minutes 1000 pieces of data are calculated and written to a text file. This file is now 250 MB and getting very difficult to deal with. It crashes when opening, things crash when it is loaded, etc. Overall it was not written well. My job is to fix it, but I am not sure where to start. My coworkers have suggested splitting the file up so that every month a new file is generated, and only the last years worth of files are loaded, unless the user needs more. Another suggestion was using a binary file instead of text. Another suggestion was to convert the text file to a database. Someone thought that Access might not be able to handle a record with 1000 pieces of data. I have a lot of research to do, but I thought that this might help me in the right direction!! Any ideas you have would be greatly appreciated. Jennifer
-
I am currently debugging a program that stores large amounts of data. Every 15 minutes 1000 pieces of data are calculated and written to a text file. This file is now 250 MB and getting very difficult to deal with. It crashes when opening, things crash when it is loaded, etc. Overall it was not written well. My job is to fix it, but I am not sure where to start. My coworkers have suggested splitting the file up so that every month a new file is generated, and only the last years worth of files are loaded, unless the user needs more. Another suggestion was using a binary file instead of text. Another suggestion was to convert the text file to a database. Someone thought that Access might not be able to handle a record with 1000 pieces of data. I have a lot of research to do, but I thought that this might help me in the right direction!! Any ideas you have would be greatly appreciated. Jennifer
-
I am currently debugging a program that stores large amounts of data. Every 15 minutes 1000 pieces of data are calculated and written to a text file. This file is now 250 MB and getting very difficult to deal with. It crashes when opening, things crash when it is loaded, etc. Overall it was not written well. My job is to fix it, but I am not sure where to start. My coworkers have suggested splitting the file up so that every month a new file is generated, and only the last years worth of files are loaded, unless the user needs more. Another suggestion was using a binary file instead of text. Another suggestion was to convert the text file to a database. Someone thought that Access might not be able to handle a record with 1000 pieces of data. I have a lot of research to do, but I thought that this might help me in the right direction!! Any ideas you have would be greatly appreciated. Jennifer
Well, if you dont want to read the whole file, you dont need to load the whole file. That is, when first loading the file, load text from the file that fills only one page of your textbox, when the user moves the scroll bars of the textbox, you should determine where to go in the text file, load text that lie in that position and display it to the user. The user does not care whether the whole text file is loaded or not, but when loading the text from the file only-as-needed, you can gain a very good performance instead of loading the whole text file. You can load a 2GB text file in milliseconds with little memory usage, little cpu processing, and (almost) no disk load. Regards, Mohammad Gdeisat أعلنت إستسلام قلبي لرحيلك..دعه يودع خطوات سيرك..فقط إجعل قلبك يحتضن حروفي ويشعرها بدفئه فقد مللت كل لحظا تي بدونك....فمابقي لي إلا أن أثني على قسوتك.... فليتك تعلم كم أنهكتني...وكم أحرقتني...وكم جعلت مني وطنا بلا معالم. فـــدعنــي أبـــكي على شواطئ دفئك...قاتلي
-
Well, if you dont want to read the whole file, you dont need to load the whole file. That is, when first loading the file, load text from the file that fills only one page of your textbox, when the user moves the scroll bars of the textbox, you should determine where to go in the text file, load text that lie in that position and display it to the user. The user does not care whether the whole text file is loaded or not, but when loading the text from the file only-as-needed, you can gain a very good performance instead of loading the whole text file. You can load a 2GB text file in milliseconds with little memory usage, little cpu processing, and (almost) no disk load. Regards, Mohammad Gdeisat أعلنت إستسلام قلبي لرحيلك..دعه يودع خطوات سيرك..فقط إجعل قلبك يحتضن حروفي ويشعرها بدفئه فقد مللت كل لحظا تي بدونك....فمابقي لي إلا أن أثني على قسوتك.... فليتك تعلم كم أنهكتني...وكم أحرقتني...وكم جعلت مني وطنا بلا معالم. فـــدعنــي أبـــكي على شواطئ دفئك...قاتلي
Mohammad A Gdeisat wrote: أعلنت إستسلام قلبي لرحيلك..دعه يودع خطوات سيرك..فقط إجعل قلبك يحتضن حروفي ويشعرها بدفئه فقد مللت كل لحظا تي بدونك....فمابقي لي إلا أن أثني على قسوتك.... فليتك تعلم كم أنهكتني...وكم أحرقتني...وكم جعلت مني وطنا بلا معالم. فـــدعنــي أبـــكي على شواطئ دفئك...قاتلي Could you please translate that in english...
"Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow
cheers, Alok Gupta VC Forum Q&A :- I/ IV
-
I am currently debugging a program that stores large amounts of data. Every 15 minutes 1000 pieces of data are calculated and written to a text file. This file is now 250 MB and getting very difficult to deal with. It crashes when opening, things crash when it is loaded, etc. Overall it was not written well. My job is to fix it, but I am not sure where to start. My coworkers have suggested splitting the file up so that every month a new file is generated, and only the last years worth of files are loaded, unless the user needs more. Another suggestion was using a binary file instead of text. Another suggestion was to convert the text file to a database. Someone thought that Access might not be able to handle a record with 1000 pieces of data. I have a lot of research to do, but I thought that this might help me in the right direction!! Any ideas you have would be greatly appreciated. Jennifer
Jenleonard wrote:
Another suggestion was to convert the text file to a database. Someone thought that Access might not be able to handle a record with 1000 pieces of data.
As memory serves Access is good till about a gig in physical size. I have a few hundred thousand entries in an access database, and it runs without a problem. It all depends on the design. Using access to start out with gives you a good reference point to scale it to a SQL server.
DEBUGGING : Removing the needles from the haystack.
-
Mohammad A Gdeisat wrote: أعلنت إستسلام قلبي لرحيلك..دعه يودع خطوات سيرك..فقط إجعل قلبك يحتضن حروفي ويشعرها بدفئه فقد مللت كل لحظا تي بدونك....فمابقي لي إلا أن أثني على قسوتك.... فليتك تعلم كم أنهكتني...وكم أحرقتني...وكم جعلت مني وطنا بلا معالم. فـــدعنــي أبـــكي على شواطئ دفئك...قاتلي Could you please translate that in english...
"Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow
cheers, Alok Gupta VC Forum Q&A :- I/ IV
I am sorry, my signature is in arabic, and it has nothing with the answer, I hope that my answer to the post was submitted correctly, here what I wrote: Well, if you dont want to read the whole file, you dont need to load the whole file. That is, when first loading the file, load text from the file that fills only one page of your textbox, when the user moves the scroll bars of the textbox, you should determine where to go in the text file, load text that lie in that position and display it to the user. The user does not care whether the whole text file is loaded or not, but when loading the text from the file only-as-needed, you can gain a very good performance instead of loading the whole text file. You can load a 2GB text file in milliseconds with little memory usage, little cpu processing, and (almost) no disk load. Regards, Mohammad Gdeisat
-
I am sorry, my signature is in arabic, and it has nothing with the answer, I hope that my answer to the post was submitted correctly, here what I wrote: Well, if you dont want to read the whole file, you dont need to load the whole file. That is, when first loading the file, load text from the file that fills only one page of your textbox, when the user moves the scroll bars of the textbox, you should determine where to go in the text file, load text that lie in that position and display it to the user. The user does not care whether the whole text file is loaded or not, but when loading the text from the file only-as-needed, you can gain a very good performance instead of loading the whole text file. You can load a 2GB text file in milliseconds with little memory usage, little cpu processing, and (almost) no disk load. Regards, Mohammad Gdeisat
Mohammad A Gdeisat wrote:
my signature is in arabic, and it has nothing with the answer
ohh you taking me wrong.. I just want to know what exactly your signature meant.
"Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow
cheers, Alok Gupta VC Forum Q&A :- I/ IV
-
Mohammad A Gdeisat wrote:
my signature is in arabic, and it has nothing with the answer
ohh you taking me wrong.. I just want to know what exactly your signature meant.
"Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow
cheers, Alok Gupta VC Forum Q&A :- I/ IV
This is the translation of the signature: "My heart has surrendered for your leaving... let it say goodbye for your steps... just let your heart take my words and feel its warmness... because I've bored all my moments without you... there is nothing left except to praise your cruelty. I wish you know, how much you've to exhausted and burnt me, and how much you made me a building without a basis... so let me cry on the shores of your warmness...my killer" Its some kind of romantic sayings in arabic, to express feelings of a loving person... maybe my translation did not give the whole meaning, but it gives the idea. Even programmers can fall in love! And ever has it been that love knows not its own depth until the hour of separation
-
This is the translation of the signature: "My heart has surrendered for your leaving... let it say goodbye for your steps... just let your heart take my words and feel its warmness... because I've bored all my moments without you... there is nothing left except to praise your cruelty. I wish you know, how much you've to exhausted and burnt me, and how much you made me a building without a basis... so let me cry on the shores of your warmness...my killer" Its some kind of romantic sayings in arabic, to express feelings of a loving person... maybe my translation did not give the whole meaning, but it gives the idea. Even programmers can fall in love! And ever has it been that love knows not its own depth until the hour of separation