High memory usage
-
Don't load so much data into memory. What are you trying to achieve that you need so much data loading into memory from a database?
Steve Jowett ------------------------- Real programmers don't comment their code. If it was hard to write, it should be hard to read.
My main motive is to save the data in the database and then on loading the application ALL the data from the database is loaded into an array of string datatype. well it is really essential to load all the data, so i'll have to load it. what i am looking for is a method to lower the memory utilization. i tried to minimize the window and then restore it. This solves the memory problem but i want to use it as the last available option. can you(or anyone) tell me how to lower the memory(RAM) utilazation?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
-
My main motive is to save the data in the database and then on loading the application ALL the data from the database is loaded into an array of string datatype. well it is really essential to load all the data, so i'll have to load it. what i am looking for is a method to lower the memory utilization. i tried to minimize the window and then restore it. This solves the memory problem but i want to use it as the last available option. can you(or anyone) tell me how to lower the memory(RAM) utilazation?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
Have you tryed to trim strings before load them in the array?
-
Hi all. in my app i have to load one or two million of strings from the database to an array. after loading i close the database connection. i run a FOR loop and load the array but memory reaches 125 MB how to make it low?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
Hi, Working on big data sets will probably end up being terribly slow. If you can stream the data, it might be orders of magnitude faster. You haven't told anything useful so far, explain what kind of data it is, where it comes from, what you want to do with it, how long it is allowed to take, and exactly why you want to close the database. :)
Luc Pattyn
Have a look at my entry for the lean-and-mean competition; please provide comments, feedback, discussion, and don’t forget to vote for it! Thank you.
Local announcement (Antwerp region): Lange Wapper? Neen!
-
My main motive is to save the data in the database and then on loading the application ALL the data from the database is loaded into an array of string datatype. well it is really essential to load all the data, so i'll have to load it. what i am looking for is a method to lower the memory utilization. i tried to minimize the window and then restore it. This solves the memory problem but i want to use it as the last available option. can you(or anyone) tell me how to lower the memory(RAM) utilazation?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
TheMrProgrammer wrote:
well it is really essential to load all the data, so i'll have to load it.
I am not sure why you need to do this. Are you doing this to make further data fetching to be from memory and do you think it is going to be faster than querying database for required data? If you need frequent data retrieval, array won't be the right data structure as it takes linear time for search.
TheMrProgrammer wrote:
what i am looking for is a method to lower the memory utilization.
Do you have 2 million unique strings? Strings are interned by the CLR and same strings can share the memory. Have you profiled your application? Use CLR profiler and find out which area is using more memory. So hard to tell more without knowing details about your application.
Navaneeth How to use google | Ask smart questions
-
My main motive is to save the data in the database and then on loading the application ALL the data from the database is loaded into an array of string datatype. well it is really essential to load all the data, so i'll have to load it. what i am looking for is a method to lower the memory utilization. i tried to minimize the window and then restore it. This solves the memory problem but i want to use it as the last available option. can you(or anyone) tell me how to lower the memory(RAM) utilazation?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
You don't need to worry about it. What you did by minimizing the app and restoring it tells me that you only looked in TaskManager for the memory statistic. Don't. What you're seeing the memory the .NET CLR has RESERVED for your application, not how much your app is actually using. Use Performance Monitor instead to see what your app is really using. You do not need to worry about the extra memory being reserved. The .NET Memory Manager keeps extra memory in the pool to allocate new objects your app wants as fast as possible. WIthout this, the Memory Manager has to request more memory from Windows and add it to the Managed Pool, which takes time. So, yeah, you can reduce this extra pool, but you're only hurting your apps performance by doing so. Now, if Windows needs that memory back, the Memory Manager will be more than happy to return any extra memory it can back to Windows, without you doing anything.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
2006, 2007, 2008
But no longer in 2009... -
Hi, Working on big data sets will probably end up being terribly slow. If you can stream the data, it might be orders of magnitude faster. You haven't told anything useful so far, explain what kind of data it is, where it comes from, what you want to do with it, how long it is allowed to take, and exactly why you want to close the database. :)
Luc Pattyn
Have a look at my entry for the lean-and-mean competition; please provide comments, feedback, discussion, and don’t forget to vote for it! Thank you.
Local announcement (Antwerp region): Lange Wapper? Neen!
My app saves data (string) in the database which is of .mdb format. At the startup all the data from the database is loaded into the memory because the app needs to iterate through the data many times. I think it will be faster to iterate through the string array rather than iterating through the database, thats why i load it in a string array. am I correct in thinking so? I use ADODB to connect to the database. Is there a way to have the strings in the memory, without eating up the RAM? how do i convert the data into stream? well what is stream anyway?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
-
Have you tryed to trim strings before load them in the array?
if you mean using the Trim() function, there is no need because there are no spaces(leading or trailing). thanks anyway for the suggestion.
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
-
TheMrProgrammer wrote:
well it is really essential to load all the data, so i'll have to load it.
I am not sure why you need to do this. Are you doing this to make further data fetching to be from memory and do you think it is going to be faster than querying database for required data? If you need frequent data retrieval, array won't be the right data structure as it takes linear time for search.
TheMrProgrammer wrote:
what i am looking for is a method to lower the memory utilization.
Do you have 2 million unique strings? Strings are interned by the CLR and same strings can share the memory. Have you profiled your application? Use CLR profiler and find out which area is using more memory. So hard to tell more without knowing details about your application.
Navaneeth How to use google | Ask smart questions
Yes I need frequent data retrieval. can you tell me waht will be the right data structure to do so. Thanks
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
-
You don't need to worry about it. What you did by minimizing the app and restoring it tells me that you only looked in TaskManager for the memory statistic. Don't. What you're seeing the memory the .NET CLR has RESERVED for your application, not how much your app is actually using. Use Performance Monitor instead to see what your app is really using. You do not need to worry about the extra memory being reserved. The .NET Memory Manager keeps extra memory in the pool to allocate new objects your app wants as fast as possible. WIthout this, the Memory Manager has to request more memory from Windows and add it to the Managed Pool, which takes time. So, yeah, you can reduce this extra pool, but you're only hurting your apps performance by doing so. Now, if Windows needs that memory back, the Memory Manager will be more than happy to return any extra memory it can back to Windows, without you doing anything.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
2006, 2007, 2008
But no longer in 2009...hey thanks fer tha info
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
-
My app saves data (string) in the database which is of .mdb format. At the startup all the data from the database is loaded into the memory because the app needs to iterate through the data many times. I think it will be faster to iterate through the string array rather than iterating through the database, thats why i load it in a string array. am I correct in thinking so? I use ADODB to connect to the database. Is there a way to have the strings in the memory, without eating up the RAM? how do i convert the data into stream? well what is stream anyway?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
TheMrProgrammer wrote:
I think it will be faster to iterate through the string array rather than iterating through the database, thats why i load it in a string array. am I correct in thinking so?
No. A simple DB query with required filter in
wherecondition will be faster than iterating an array. As I said, in worst case, searches in arrays takes O(n) time. This becomes pretty slow when you have huge number of data. Databases are clever to handle searches and can perform well than a linear storage like array.TheMrProgrammer wrote:
because the app needs to iterate through the data many times.
If you still need data in memory, load it to a right data structure that supports faster searches. You can try a SortedList which will do searches in O(log n) time as it uses binary search internally. This can improve the search time drastically. If you have 2 million unique strings,
HashSet(T)is also a good choice. It uses hash tables internally and searches are done in constant (O(1)) time.Navaneeth How to use google | Ask smart questions
-
My app saves data (string) in the database which is of .mdb format. At the startup all the data from the database is loaded into the memory because the app needs to iterate through the data many times. I think it will be faster to iterate through the string array rather than iterating through the database, thats why i load it in a string array. am I correct in thinking so? I use ADODB to connect to the database. Is there a way to have the strings in the memory, without eating up the RAM? how do i convert the data into stream? well what is stream anyway?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
TheMrProgrammer wrote:
I think it will be faster to iterate through the string array rather than iterating through the database, thats why i load it in a string array. am I correct in thinking so?
If by iterating you mean you will need each row once, then loading them all at once is the worst thing you could do.
TheMrProgrammer wrote:
Is there a way to have the strings in the memory, without eating up the RAM?
Magic powder perhaps. :)
Luc Pattyn
Have a look at my entry for the lean-and-mean competition; please provide comments, feedback, discussion, and don’t forget to vote for it! Thank you.
Local announcement (Antwerp region): Lange Wapper? Neen!
-
Hi all. in my app i have to load one or two million of strings from the database to an array. after loading i close the database connection. i run a FOR loop and load the array but memory reaches 125 MB how to make it low?
TheMrProgrammer http://www.icbse.com/2009/funny-exam-answers-school-students http://download.cnet.com/TheCalcMan/3000-2094\_4-10958266.html
If I were you I would check into how to call garbage collection more often or call it explicitly. Also something you might find useful is the use of memory pressure. Thanks,
Humble Programmer
-
TheMrProgrammer wrote:
I think it will be faster to iterate through the string array rather than iterating through the database, thats why i load it in a string array. am I correct in thinking so?
If by iterating you mean you will need each row once, then loading them all at once is the worst thing you could do.
TheMrProgrammer wrote:
Is there a way to have the strings in the memory, without eating up the RAM?
Magic powder perhaps. :)
Luc Pattyn
Have a look at my entry for the lean-and-mean competition; please provide comments, feedback, discussion, and don’t forget to vote for it! Thank you.
Local announcement (Antwerp region): Lange Wapper? Neen!
Luc Pattyn wrote:
Magic powder perhaps.
You are wrong! Magic powder sits out of managed heap and there is no way for it to reduce memory allocated for managed strings. :)
Navaneeth How to use google | Ask smart questions