Out of Memory Exception in C++
-
Hi Everyone, I have a application which needs to read a file of size 113MB and stores each row data into a class object which is in turn stored in a std::map. Running it in 32-bit 4 GB RAM , the "Out of Memory" error triggered. After a observation , I felt that the RAM is not sufficient, so now I tried it in 64-bit 8GB RAM system but still the same exception is thrown. Can anyone help me in finding what exactly might be problem and how it can be overcome? Thanks in Advance!!
divya03 wrote:
I tried it in 64-bit 8GB RAM system but still the same exception is thrown
Did you rebuild for a 64 bit target?
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment "Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst "I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
-
divya03 wrote:
I tried it in 64-bit 8GB RAM system but still the same exception is thrown
Did you rebuild for a 64 bit target?
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment "Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst "I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
-
I understand that much from your initial post. But does it all need to me in memory concurrently? It's highly unlikely that it does, but only you can be the ultimate judge of that. What exactly are you doing to/with the data? Maybe an on-disk solution can be suggested.
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"You can easily judge the character of a man by how he treats those who can do nothing for him." - James D. Miles
Thank you for the suggestion. I could have tried that if all data were not required and could have read only the required data from the file and store it in memory. But here the need is, after loading of file, it shall take inputs from user either to process all the data or required data, if its all data then it has to have all the data stored in memory.
-
Perhaps there is something useful in this thread[^]. Do you have an estimate of the amount of memory needed?
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment "Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst "I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
-
Hi Everyone, I have a application which needs to read a file of size 113MB and stores each row data into a class object which is in turn stored in a std::map. Running it in 32-bit 4 GB RAM , the "Out of Memory" error triggered. After a observation , I felt that the RAM is not sufficient, so now I tried it in 64-bit 8GB RAM system but still the same exception is thrown. Can anyone help me in finding what exactly might be problem and how it can be overcome? Thanks in Advance!!
118Mb isn't all that big. You could easily hold it in memory. The map is costing you the size of the class plus about four pointers worth of storage for each entry. Perhaps your class stores the data very inefficiently. Perhaps you have a bug on reading in the data, like not recognizing end of file and adding endless entries until you run out of memory. Perhaps you have a memory leak. * How many rows are in the file? You can easily build a function to count rows using the current function that inputs data, but without saving the data. * How many rows are read in at the time the out-of-memory exception occurs? This and the last question might tell you if you had a bug inputting the data. * Are you using std::string to hold any of the row data, because memory costs can really add up when using std::string. Maybe your representation is too expensive for the memory you have. You could approach this problem by reading in the file as a block, and building a data structure that puts pointers into the file block for each line, and then parse out a single line when you need one.
-
Hi Everyone, I have a application which needs to read a file of size 113MB and stores each row data into a class object which is in turn stored in a std::map. Running it in 32-bit 4 GB RAM , the "Out of Memory" error triggered. After a observation , I felt that the RAM is not sufficient, so now I tried it in 64-bit 8GB RAM system but still the same exception is thrown. Can anyone help me in finding what exactly might be problem and how it can be overcome? Thanks in Advance!!
Are you sure the memory is actually the issue and you aren't hitting the class object limit. From memory the allocation just starts failing when you hit the 10,000 limit. Adding more memory doesn't help because it isn't the problem in the first place, you just can't allocate any more objects. Easy to check count objects as you create them and see what count you get to.
In vino veritas
-
118Mb isn't all that big. You could easily hold it in memory. The map is costing you the size of the class plus about four pointers worth of storage for each entry. Perhaps your class stores the data very inefficiently. Perhaps you have a bug on reading in the data, like not recognizing end of file and adding endless entries until you run out of memory. Perhaps you have a memory leak. * How many rows are in the file? You can easily build a function to count rows using the current function that inputs data, but without saving the data. * How many rows are read in at the time the out-of-memory exception occurs? This and the last question might tell you if you had a bug inputting the data. * Are you using std::string to hold any of the row data, because memory costs can really add up when using std::string. Maybe your representation is too expensive for the memory you have. You could approach this problem by reading in the file as a block, and building a data structure that puts pointers into the file block for each line, and then parse out a single line when you need one.
-
Are you sure the memory is actually the issue and you aren't hitting the class object limit. From memory the allocation just starts failing when you hit the 10,000 limit. Adding more memory doesn't help because it isn't the problem in the first place, you just can't allocate any more objects. Easy to check count objects as you create them and see what count you get to.
In vino veritas
-
Total number of rows are around 7 lakhs. The out of memory is occurring after around 4lakhs rows are read and when the memory in task manager shows around 700000k by the application. Yes std:string is used as a member variable of class object.
On average, std::string allocates enough memory to hold 1.5x the size of the actual string. (This is a rule of thumb), plus 2 pointers of overhead. This could be your problem, depending how many strings you have per row. Depending how you've written your data structure, it's quite likely that you have really exhausted memory. You need to rethink the problem so to solve it using less memory.
-
Hi Everyone, I have a application which needs to read a file of size 113MB and stores each row data into a class object which is in turn stored in a std::map. Running it in 32-bit 4 GB RAM , the "Out of Memory" error triggered. After a observation , I felt that the RAM is not sufficient, so now I tried it in 64-bit 8GB RAM system but still the same exception is thrown. Can anyone help me in finding what exactly might be problem and how it can be overcome? Thanks in Advance!!