Looking for ideas on how to release memory right before a generation 3 garbage collection [modified]
-
This is a question for GC experts out there: [Please don't respond telling me to use Dispose, it's not what I'm looking for.] I'm currently looking for ideas on how to release memory right before a generation 3 garbage collection. Specifically, I have a bunch of objects that all hold the contents of corresponding files on disk. I'd like to hold as many objects as possible in RAM and only release them as more RAM is needed. The naive approach is to use a Dictionary of WeakReferences. This is what I'm currently doing, but the problem is that some objects never get out of generation 0, and thus are collected and then re-loaded a few seconds later. So are there any ideas with regard to getting a good estimate of when a garbage collection is coming so I can move unused objects into WeakReferences? Some constraints: 1 - My deployment scenario is Mono + Linux, so I can't use the Win32 API, hidden CLR functions, COM, ect, ect. 2 - I'm developing with Visual Studio 2005 Pro. I can't afford VS 2008 Pro, and 2005 has some vital threading features in the debugger that VS 2008 express lacks.
modified on Friday, December 4, 2009 5:32 AM
Best thing is to use dispose() pattern to release memory by using below code dispose() { base.dispose(); GC.SuppressFinalize(this) }
-
Best thing is to use dispose() pattern to release memory by using below code dispose() { base.dispose(); GC.SuppressFinalize(this) }
-
This has nothing to do with Dispose. I'm trying to release references right before a garbage collection occurs. It's for a cache that should grow to occupy as much RAM as possible.
Pls. refers once purpose of Dispose pattern and supressfinalize method of GC.
-
Pls. refers once purpose of Dispose pattern and supressfinalize method of GC.
Dispose isn't what I'm looking for. I fully understand the pattern, and it's not appropriate in this situation. I already described why it won't work here: http://www.codeproject.com/Messages/3293314/Re-Looking-for-ideas-on-how-to-release-memory-righ.aspx[^]
-
Dispose isn't what I'm looking for. I fully understand the pattern, and it's not appropriate in this situation. I already described why it won't work here: http://www.codeproject.com/Messages/3293314/Re-Looking-for-ideas-on-how-to-release-memory-righ.aspx[^]
Kindly change the question and state the correct problem there......
-
Kindly change the question and state the correct problem there......
-
Kindly change the question and state the correct problem there......
-
This is a question for GC experts out there: [Please don't respond telling me to use Dispose, it's not what I'm looking for.] I'm currently looking for ideas on how to release memory right before a generation 3 garbage collection. Specifically, I have a bunch of objects that all hold the contents of corresponding files on disk. I'd like to hold as many objects as possible in RAM and only release them as more RAM is needed. The naive approach is to use a Dictionary of WeakReferences. This is what I'm currently doing, but the problem is that some objects never get out of generation 0, and thus are collected and then re-loaded a few seconds later. So are there any ideas with regard to getting a good estimate of when a garbage collection is coming so I can move unused objects into WeakReferences? Some constraints: 1 - My deployment scenario is Mono + Linux, so I can't use the Win32 API, hidden CLR functions, COM, ect, ect. 2 - I'm developing with Visual Studio 2005 Pro. I can't afford VS 2008 Pro, and 2005 has some vital threading features in the debugger that VS 2008 express lacks.
modified on Friday, December 4, 2009 5:32 AM
AFAICT, there is no way to tell when a GC is going to occur without diving into the CLR-specific API. Short of creating your own version of the CLR, I don't see how you can do it.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
2006, 2007, 2008
But no longer in 2009... -
AFAICT, there is no way to tell when a GC is going to occur without diving into the CLR-specific API. Short of creating your own version of the CLR, I don't see how you can do it.
A guide to posting questions on CodeProject[^]
Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
2006, 2007, 2008
But no longer in 2009...Yeah, I agree. The techniques described for .Net 3.0 seem a bit sketchy. I ended up using an algorithm that checks GC.GetTotalMemory() on a periodic basis. As a consequence, the system administrator will have to tune the program to give it a target RAM amount to occupy.
-
This is a question for GC experts out there: [Please don't respond telling me to use Dispose, it's not what I'm looking for.] I'm currently looking for ideas on how to release memory right before a generation 3 garbage collection. Specifically, I have a bunch of objects that all hold the contents of corresponding files on disk. I'd like to hold as many objects as possible in RAM and only release them as more RAM is needed. The naive approach is to use a Dictionary of WeakReferences. This is what I'm currently doing, but the problem is that some objects never get out of generation 0, and thus are collected and then re-loaded a few seconds later. So are there any ideas with regard to getting a good estimate of when a garbage collection is coming so I can move unused objects into WeakReferences? Some constraints: 1 - My deployment scenario is Mono + Linux, so I can't use the Win32 API, hidden CLR functions, COM, ect, ect. 2 - I'm developing with Visual Studio 2005 Pro. I can't afford VS 2008 Pro, and 2005 has some vital threading features in the debugger that VS 2008 express lacks.
modified on Friday, December 4, 2009 5:32 AM
Idea: Maintain a queue of strong references to the most recently used objects. This prevents them from being garbage collected. When an object is used, move it to the front of the queue. When you insert a new object at the front of the queue, delete the strong reference at the tail of the queue if the queue size is above a certain threshold. If recently-referenced objects are more likely to be referenced again, this cache will avoid re-reading them from disk. An enhancement: Instead of just a plain queue, keep the items in keyed storage (hash table, binary tree, etc), and maintain the queue as a doubly-linked list, implemented by Previous/Next fields in your items. This will let you quickly look up an object which may be in your cache. Adjust the Previous/Next fields to move it to the head of the queue. You'll also want to maintain external pointers to the head and tail of the queue. Hope this helps.
-
This is a question for GC experts out there: [Please don't respond telling me to use Dispose, it's not what I'm looking for.] I'm currently looking for ideas on how to release memory right before a generation 3 garbage collection. Specifically, I have a bunch of objects that all hold the contents of corresponding files on disk. I'd like to hold as many objects as possible in RAM and only release them as more RAM is needed. The naive approach is to use a Dictionary of WeakReferences. This is what I'm currently doing, but the problem is that some objects never get out of generation 0, and thus are collected and then re-loaded a few seconds later. So are there any ideas with regard to getting a good estimate of when a garbage collection is coming so I can move unused objects into WeakReferences? Some constraints: 1 - My deployment scenario is Mono + Linux, so I can't use the Win32 API, hidden CLR functions, COM, ect, ect. 2 - I'm developing with Visual Studio 2005 Pro. I can't afford VS 2008 Pro, and 2005 has some vital threading features in the debugger that VS 2008 express lacks.
modified on Friday, December 4, 2009 5:32 AM
It's your app, so you know if this is possible, but I would try to resturcture the logic so I do not need to be accessing the same file repeatedly to the point where I'm keeping a cache of files. If you are only accessing a file once every 10 minutes, I would assume that the performance gain from caching the string will be small. If the Mono garbage collector works like the .NET one, the Large Object Heap may be what you need. (I do not claim to be a Large Object expert. I've only read the first article that comes up on Google about it. [1][2]) If the resulting string from the file is >85K in size, the string should get placed on the LOH and will thus not get collected except in a full collection. If the files are smaller than that and you still need to cache them, you may be able to hack something together with a large finalizable object, but I wouldn't recommend it. If that doesn't work, you probably need a more defined caching policy than "keep them as long as the process has memory."
-
Idea: Maintain a queue of strong references to the most recently used objects. This prevents them from being garbage collected. When an object is used, move it to the front of the queue. When you insert a new object at the front of the queue, delete the strong reference at the tail of the queue if the queue size is above a certain threshold. If recently-referenced objects are more likely to be referenced again, this cache will avoid re-reading them from disk. An enhancement: Instead of just a plain queue, keep the items in keyed storage (hash table, binary tree, etc), and maintain the queue as a doubly-linked list, implemented by Previous/Next fields in your items. This will let you quickly look up an object which may be in your cache. Adjust the Previous/Next fields to move it to the head of the queue. You'll also want to maintain external pointers to the head and tail of the queue. Hope this helps.
That's pretty much what I decided to implement, although I'm experimenting with GC.GetTotalMemory() to determine how many items to pull off of the queue. I keep building the queue until GC.GetTotalMemory() reaches a target size, and if GC.GetTotalMemory() goes above a certain threshold, I remove two or three items. I'm not quite sure how the linked list plays in: I'm keeping references with a dictionary, and the queue is used to increment / decrement an access count. Anyway, thanks!
-
It's your app, so you know if this is possible, but I would try to resturcture the logic so I do not need to be accessing the same file repeatedly to the point where I'm keeping a cache of files. If you are only accessing a file once every 10 minutes, I would assume that the performance gain from caching the string will be small. If the Mono garbage collector works like the .NET one, the Large Object Heap may be what you need. (I do not claim to be a Large Object expert. I've only read the first article that comes up on Google about it. [1][2]) If the resulting string from the file is >85K in size, the string should get placed on the LOH and will thus not get collected except in a full collection. If the files are smaller than that and you still need to cache them, you may be able to hack something together with a large finalizable object, but I wouldn't recommend it. If that doesn't work, you probably need a more defined caching policy than "keep them as long as the process has memory."
Thanks, but this is for a web server. (I'm also caching more then just text files, but I had to give a simplified example.) The program needs to dynamically adapt to whatever becomes popular without manual tuning from a system administrator. I ended up doing something similar to what was suggested here: http://www.codeproject.com/Messages/3293896/Re-Looking-for-ideas-on-how-to-release-memory-righ.aspx[^]
-
This is a question for GC experts out there: [Please don't respond telling me to use Dispose, it's not what I'm looking for.] I'm currently looking for ideas on how to release memory right before a generation 3 garbage collection. Specifically, I have a bunch of objects that all hold the contents of corresponding files on disk. I'd like to hold as many objects as possible in RAM and only release them as more RAM is needed. The naive approach is to use a Dictionary of WeakReferences. This is what I'm currently doing, but the problem is that some objects never get out of generation 0, and thus are collected and then re-loaded a few seconds later. So are there any ideas with regard to getting a good estimate of when a garbage collection is coming so I can move unused objects into WeakReferences? Some constraints: 1 - My deployment scenario is Mono + Linux, so I can't use the Win32 API, hidden CLR functions, COM, ect, ect. 2 - I'm developing with Visual Studio 2005 Pro. I can't afford VS 2008 Pro, and 2005 has some vital threading features in the debugger that VS 2008 express lacks.
modified on Friday, December 4, 2009 5:32 AM
Hi, Did you think about implementing a paging algorithm (LRU, clock page replacement) ? As for the dispose, I recommend that in your dispose you set the memebers to null where possible for faster memory reclaims.
-
Hi, Did you think about implementing a paging algorithm (LRU, clock page replacement) ? As for the dispose, I recommend that in your dispose you set the memebers to null where possible for faster memory reclaims.