Large Heap Doesn't Compact?
-
I have written a C# application that uses very large objects: 1mb to 3mb, it has what seems to be a memory leak. According to microsoft and all my colleague .NET Applications DON'T leak. So I looked for an answer and came across that: "The large object heap is never compacted..." http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/DBGch02.asp[^] a quote from microsoft article about memory debugging. Isn't that means that when a memory is freed a hole is created and not handled? Is that the reason my app is leaking? Thanks b.v.
BarV wrote:
According to microsoft and all my colleague .NET Applications DON'T leak.
True, to a point. A .NET app can only leak unmanaged resources or memory allocated through unmanaged means.
BarV wrote:
Is that the reason my app is leaking?
What makes you think your app is leaking? Don't tell me you relied soley on the numbers you saw in TaskManager?
Dave Kreskowiak Microsoft MVP - Visual Basic
-
I have written a C# application that uses very large objects: 1mb to 3mb, it has what seems to be a memory leak. According to microsoft and all my colleague .NET Applications DON'T leak. So I looked for an answer and came across that: "The large object heap is never compacted..." http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/DBGch02.asp[^] a quote from microsoft article about memory debugging. Isn't that means that when a memory is freed a hole is created and not handled? Is that the reason my app is leaking? Thanks b.v.
BarV wrote:
Isn't that means that when a memory is freed a hole is created and not handled?
No, it doesn't. The hole is created, all right, as the heap is not compacted, but it's handled. The memory will be used for allocating new large objects.
--- b { font-weight: normal; }
-
BarV wrote:
According to microsoft and all my colleague .NET Applications DON'T leak.
True, to a point. A .NET app can only leak unmanaged resources or memory allocated through unmanaged means.
BarV wrote:
Is that the reason my app is leaking?
What makes you think your app is leaking? Don't tell me you relied soley on the numbers you saw in TaskManager?
Dave Kreskowiak Microsoft MVP - Visual Basic
-
BarV wrote:
Isn't that means that when a memory is freed a hole is created and not handled?
No, it doesn't. The hole is created, all right, as the heap is not compacted, but it's handled. The memory will be used for allocating new large objects.
--- b { font-weight: normal; }
-
No, it's not. Moving large blocks of memory around can take an eternity, so there is no compaction, or defragmentation as your calling it. The hole remains where it was created. But the CLR is smart enough to fill that hole with whatever it can, whenever it can. The memory is not "just wasted". The memory stays allocated as far as an unmanaged view of the memory goes. Which is what you saw when you used the Process' Private Bytes counter. The memory is not used by your application, as far as Windows is concerned, it is. This is because the memory is reserved by the CLR in the Managed Heap. You MUST used the .NET CLR performance object counters if you want to see how much memory your app is REALLY using. What you're looking at, when using the Private Bytes counter, is the memory reserved for your application (allocated by your app or not) by the .NET CLR virtual machine your app is running in. -- modified at 15:20 Monday 21st August, 2006
Dave Kreskowiak Microsoft MVP - Visual Basic
-
No, it's not. Moving large blocks of memory around can take an eternity, so there is no compaction, or defragmentation as your calling it. The hole remains where it was created. But the CLR is smart enough to fill that hole with whatever it can, whenever it can. The memory is not "just wasted". The memory stays allocated as far as an unmanaged view of the memory goes. Which is what you saw when you used the Process' Private Bytes counter. The memory is not used by your application, as far as Windows is concerned, it is. This is because the memory is reserved by the CLR in the Managed Heap. You MUST used the .NET CLR performance object counters if you want to see how much memory your app is REALLY using. What you're looking at, when using the Private Bytes counter, is the memory reserved for your application (allocated by your app or not) by the .NET CLR virtual machine your app is running in. -- modified at 15:20 Monday 21st August, 2006
Dave Kreskowiak Microsoft MVP - Visual Basic
-
No, it's not. Moving large blocks of memory around can take an eternity, so there is no compaction, or defragmentation as your calling it. The hole remains where it was created. But the CLR is smart enough to fill that hole with whatever it can, whenever it can. The memory is not "just wasted". The memory stays allocated as far as an unmanaged view of the memory goes. Which is what you saw when you used the Process' Private Bytes counter. The memory is not used by your application, as far as Windows is concerned, it is. This is because the memory is reserved by the CLR in the Managed Heap. You MUST used the .NET CLR performance object counters if you want to see how much memory your app is REALLY using. What you're looking at, when using the Private Bytes counter, is the memory reserved for your application (allocated by your app or not) by the .NET CLR virtual machine your app is running in. -- modified at 15:20 Monday 21st August, 2006
Dave Kreskowiak Microsoft MVP - Visual Basic
Dave Kreskowiak wrote:
No, it's not. Moving large blocks of memory around can take an eternity, so there is no compaction, or defragmentation as your calling it. The hole remains where it was created. But the CLR is smart enough to fill that hole with whatever it can, whenever it can. The memory is not "just wasted".
Not in bulk, but in small amounts, it can effectively be. The problem is that the large object heap can fragment in the same way as a standard heap in a native app. The risk of large scale fragementation is minimal though because you need frequent de/allocation on the heap to cause it, and you shouldn't be creating and destroying large objects very often.
-
Dave Kreskowiak wrote:
No, it's not. Moving large blocks of memory around can take an eternity, so there is no compaction, or defragmentation as your calling it. The hole remains where it was created. But the CLR is smart enough to fill that hole with whatever it can, whenever it can. The memory is not "just wasted".
Not in bulk, but in small amounts, it can effectively be. The problem is that the large object heap can fragment in the same way as a standard heap in a native app. The risk of large scale fragementation is minimal though because you need frequent de/allocation on the heap to cause it, and you shouldn't be creating and destroying large objects very often.
Various pieces of a strcutre or class can be rearranged, but not something like an entire array. I've worked with HUGE arrays (250+MB) and those don't get compacted or moved. The holes they created stay in place until another array was allocated or an existing array expanded. Because the indexer is required to be contiguous, moving an array of that size after it's been swapped to disk can take forever and a day as far as your app is concerned.
Dave Kreskowiak Microsoft MVP - Visual Basic
-
Various pieces of a strcutre or class can be rearranged, but not something like an entire array. I've worked with HUGE arrays (250+MB) and those don't get compacted or moved. The holes they created stay in place until another array was allocated or an existing array expanded. Because the indexer is required to be contiguous, moving an array of that size after it's been swapped to disk can take forever and a day as far as your app is concerned.
Dave Kreskowiak Microsoft MVP - Visual Basic
Dave Kreskowiak wrote:
Various pieces of a strcutre or class can be rearranged, but not something like an entire array. I've worked with HUGE arrays (250+MB) and those don't get compacted or moved. The holes they created stay in place until another array was allocated or an existing array expanded.
That was my point, which is how in theory the heap can fragment. The following psudeocode will put a 10byte fragment on the large object heapthat will remain unused for the remainder of the app's runtime. Allocate(TempObject, 1000) // in use, bytes 1-1000 Allocate(PremamantObject1, 1000) // in use, bytes 1-2000 Deallocate(TempObject) // in use, bytes 1001-2000 Allocate(PremamantObject2, 990) // in use, bytes 1-990, 1001-2000. //Unallocated, and too small to use bytes 991-1000 In a well designed app, this won't happen often enough to be a problem, but with a sloppy design that's continually re and deallocating from the large object heap you could end up with a number of such fragments around objects that have long lifetimes. Worst case could be almost 50% loss with a sequence of minimum sized objects, and gaps one byte smaller than the minimum. If this is happening there's bad design to blame, but a heap fragmentation leak's possible in theory, which was my point.
-
Dave Kreskowiak wrote:
Various pieces of a strcutre or class can be rearranged, but not something like an entire array. I've worked with HUGE arrays (250+MB) and those don't get compacted or moved. The holes they created stay in place until another array was allocated or an existing array expanded.
That was my point, which is how in theory the heap can fragment. The following psudeocode will put a 10byte fragment on the large object heapthat will remain unused for the remainder of the app's runtime. Allocate(TempObject, 1000) // in use, bytes 1-1000 Allocate(PremamantObject1, 1000) // in use, bytes 1-2000 Deallocate(TempObject) // in use, bytes 1001-2000 Allocate(PremamantObject2, 990) // in use, bytes 1-990, 1001-2000. //Unallocated, and too small to use bytes 991-1000 In a well designed app, this won't happen often enough to be a problem, but with a sloppy design that's continually re and deallocating from the large object heap you could end up with a number of such fragments around objects that have long lifetimes. Worst case could be almost 50% loss with a sequence of minimum sized objects, and gaps one byte smaller than the minimum. If this is happening there's bad design to blame, but a heap fragmentation leak's possible in theory, which was my point.
Oh! I know how the heap can fragment! The trick is knowing how to defrag it yourself. The app with the HUGE arrays allocated two arrays on the LOH. This app iterated over a set of data, over and over again, until a solution was found. The first array was the data from the last pass and the second was the data generated from the current pass. Well, pass after pass, the LOH grew and grew, faster than the amount of data that was generated. Weird - and very much like the OP's problem. The solution to implementing LOH compaction in this case was surprisingly easy. Since I only had two objects only needed one of them the next pass, I simply serialized the generated array to disk, disposed of both arrays, forced a GC, then deserialized the array from disk to put it at the bottom of the LOH. A poor-man's memory manager, if you will. According to the CLR Profiler and a bunch of testing, it worked beautifully! It's not the best performing solution, but I didn't need the app to respond between passes. Most of the apps time was spent crunching on that ever growing pile of data... Now, implementing something similar in the OP's case depends on his objects and requirements...
Dave Kreskowiak Microsoft MVP - Visual Basic