small, slow memory leak
-
is the most annoying bug to address. I think I may lose my mind today. I've never written an application that suffered from one, and I've written numerous applications similar to this current one. I fear it may be due to my interaction with a legacy set of libraries.
-
is the most annoying bug to address. I think I may lose my mind today. I've never written an application that suffered from one, and I've written numerous applications similar to this current one. I fear it may be due to my interaction with a legacy set of libraries.
We are talking about C++, I presume. It has been a while, but I remember using debug libraries to monitor memory allocation. They were included with Visual Studio, but I just can't remember the name... Edit: Got it! You must include crtdbg.h and, of course use the functions from the library to monitor memory allocation.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverb -
We are talking about C++, I presume. It has been a while, but I remember using debug libraries to monitor memory allocation. They were included with Visual Studio, but I just can't remember the name... Edit: Got it! You must include crtdbg.h and, of course use the functions from the library to monitor memory allocation.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverbActually C#. It's hard to debug when it is literally slowly crawling up. Let's say 300k over 30 seconds, but it comes down, too. Just that over time it's growing faster than it's shrinking. I didn't mean to pose it as a programming question, just an observation that slow small leaks are the worst!
-
is the most annoying bug to address. I think I may lose my mind today. I've never written an application that suffered from one, and I've written numerous applications similar to this current one. I fear it may be due to my interaction with a legacy set of libraries.
Yeah, I'm dealing with something similar here... A third-party library (Which I have no choice but to use) is leaking 4KB every time I do a certain type of call through it (You have no idea how difficult it was to track down that leak)... I recently had to add some functionality to my system, which now needs to perform this call an average of once every 3-4 seconds. So my server, previously holding steady at about 80-90MB of RAM (Hey, it does quite a bit!), now increases by ~3.4MB/hour, or 80MB/day. Hated to have to do it, but as of today, my server process now restarts itself once a day to mitigate this :(
Proud to have finally moved to the A-Ark. Which one are you in?
Author of the Guardians Saga (Sci-Fi/Fantasy novels) -
Actually C#. It's hard to debug when it is literally slowly crawling up. Let's say 300k over 30 seconds, but it comes down, too. Just that over time it's growing faster than it's shrinking. I didn't mean to pose it as a programming question, just an observation that slow small leaks are the worst!
I'd love to have the code mem leaks - luv them!
-
is the most annoying bug to address. I think I may lose my mind today. I've never written an application that suffered from one, and I've written numerous applications similar to this current one. I fear it may be due to my interaction with a legacy set of libraries.
I remember using windbg and sos the last time I tracked something like that down in managed code. Just get a dump of all active objects and you should be able to figure it out. If it is using unmanaged resources then good luck!
Curvature of the Mind now with 3D
-
Yeah, I'm dealing with something similar here... A third-party library (Which I have no choice but to use) is leaking 4KB every time I do a certain type of call through it (You have no idea how difficult it was to track down that leak)... I recently had to add some functionality to my system, which now needs to perform this call an average of once every 3-4 seconds. So my server, previously holding steady at about 80-90MB of RAM (Hey, it does quite a bit!), now increases by ~3.4MB/hour, or 80MB/day. Hated to have to do it, but as of today, my server process now restarts itself once a day to mitigate this :(
Proud to have finally moved to the A-Ark. Which one are you in?
Author of the Guardians Saga (Sci-Fi/Fantasy novels)That is no fun. Do you have to fully restart, or can you clean it up, sort of reinitialize? I have had coworkers do similar things to get things by and into beta. Restarting weekly. (in my industry nothing happens on Saturdays, ever) I also just realized this issue only happens on the production server facing the production database. Running it on my corporate dev box, against the dev database produces level memory. It is subscribed to the exact same feed of live data. The prod server is running Windows Server SP1 and my desktop is running Windows 7. I've also seen an allusive error on the prod box that has never happened on my corporate box running the same program simultaneously.
-
I'd love to have the code mem leaks - luv them!
-
Actually C#. It's hard to debug when it is literally slowly crawling up. Let's say 300k over 30 seconds, but it comes down, too. Just that over time it's growing faster than it's shrinking. I didn't mean to pose it as a programming question, just an observation that slow small leaks are the worst!
I worked on an application that puts together huge download files from a database. Even the delay caused by the garbage collection may be too much. First, look that you implement (and use) Dispose() on all relevant large data objects. Without having to wait for garbage collection the real behavior of your application gets much clearer to see. And if there really is a leak, then look out for objects which hold references to each other or in a circle. Those would survive garbage collection, so it would be better to set all references to other objects to null whuile disposing them. We had a tricky situation in an ASP .Net app, where some fool stored controls in the session state. The page held references to the controls, the controls held references to the page. They were never released by the garbage collection and memory got fuller with every page request.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverb -
That is no fun. Do you have to fully restart, or can you clean it up, sort of reinitialize? I have had coworkers do similar things to get things by and into beta. Restarting weekly. (in my industry nothing happens on Saturdays, ever) I also just realized this issue only happens on the production server facing the production database. Running it on my corporate dev box, against the dev database produces level memory. It is subscribed to the exact same feed of live data. The prod server is running Windows Server SP1 and my desktop is running Windows 7. I've also seen an allusive error on the prod box that has never happened on my corporate box running the same program simultaneously.
Nah, the leak is in a Java library accessed via RCW... I don't want to put it into its own application domain, as that'd kill my performance... So the only way to release the memory is to restart the process. Granted, I already have an infrastructure in place to do just that (I can restart the service remotely, by entering a password into the client app)... I just hate relying on the old "Have you tried restarting?" tech support fix.
Proud to have finally moved to the A-Ark. Which one are you in?
Author of the Guardians Saga (Sci-Fi/Fantasy novels) -
is the most annoying bug to address. I think I may lose my mind today. I've never written an application that suffered from one, and I've written numerous applications similar to this current one. I fear it may be due to my interaction with a legacy set of libraries.
wizardzz wrote:
I fear it may be due to my interaction with a legacy set of libraries.
Yes, the legacy libraries do leak memory while attempting to escape from your code... ;P
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler. -- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong. -- Iain Clarke
[My articles] -
I worked on an application that puts together huge download files from a database. Even the delay caused by the garbage collection may be too much. First, look that you implement (and use) Dispose() on all relevant large data objects. Without having to wait for garbage collection the real behavior of your application gets much clearer to see. And if there really is a leak, then look out for objects which hold references to each other or in a circle. Those would survive garbage collection, so it would be better to set all references to other objects to null whuile disposing them. We had a tricky situation in an ASP .Net app, where some fool stored controls in the session state. The page held references to the controls, the controls held references to the page. They were never released by the garbage collection and memory got fuller with every page request.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverbCDP1802 wrote:
look out for objects which hold references to each other or in a circle. Those would survive garbage collection
Not correct. They would all survive if at least one of them is really alive, otherwise the cluster dies as it should. :)
Luc Pattyn [Forum Guidelines] [My Articles] Nil Volentibus Arduum
Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at the code.
-
CDP1802 wrote:
look out for objects which hold references to each other or in a circle. Those would survive garbage collection
Not correct. They would all survive if at least one of them is really alive, otherwise the cluster dies as it should. :)
Luc Pattyn [Forum Guidelines] [My Articles] Nil Volentibus Arduum
Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at the code.
Why? Each member of the circle is still being referenced by another member.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverb -
Why? Each member of the circle is still being referenced by another member.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverba typical GC works its way through the maze of objects, starting at the ones it knows are alive: references found on the stacks (and in CPU registers) and the static objects; then it looks for references inside those objects, recursively. Those are the only ones that could ever be accessed again. Two objects that reference each other but are floating in space, detached from stacks and statics, cannot be accessed, you don't have a reference to access them. Here is a little test you might want to run to convince yourself empirically:
public class Floater {
private Floater other;
private static long count;public Floater() { count++; if(count&0xFFFFFF)==0) Console.WriteLine("Already created "+count+" Floaters"); } public static void Link(Floater f1, Floater f2) { f1.other=f2; f2.other=f1; }}
public class Test {
public static void Test() {
for(;;) Floater.Link(new Floater(), new Floater());
}
}Now run Test.Test() and wonder why it never runs out of memory. :)
Luc Pattyn [Forum Guidelines] [My Articles] Nil Volentibus Arduum
Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at the code.
-
Why? Each member of the circle is still being referenced by another member.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverbThe circle has to be reachable from the stack or a root object. If .net used reference counting then you could have that issue.
Curvature of the Mind now with 3D
-
The circle has to be reachable from the stack or a root object. If .net used reference counting then you could have that issue.
Curvature of the Mind now with 3D
Interesting. Tomorrow at work I will get an unfixed version of a leaky application out of source control and try to find out what actually caused the leak. Obviously wrong reasoning brought me to the right answer back then.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverb -
a typical GC works its way through the maze of objects, starting at the ones it knows are alive: references found on the stacks (and in CPU registers) and the static objects; then it looks for references inside those objects, recursively. Those are the only ones that could ever be accessed again. Two objects that reference each other but are floating in space, detached from stacks and statics, cannot be accessed, you don't have a reference to access them. Here is a little test you might want to run to convince yourself empirically:
public class Floater {
private Floater other;
private static long count;public Floater() { count++; if(count&0xFFFFFF)==0) Console.WriteLine("Already created "+count+" Floaters"); } public static void Link(Floater f1, Floater f2) { f1.other=f2; f2.other=f1; }}
public class Test {
public static void Test() {
for(;;) Floater.Link(new Floater(), new Floater());
}
}Now run Test.Test() and wonder why it never runs out of memory. :)
Luc Pattyn [Forum Guidelines] [My Articles] Nil Volentibus Arduum
Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at the code.
Thanks. Looks like we fixed bugs by actually getting some things wrong :)
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverb -
Interesting. Tomorrow at work I will get an unfixed version of a leaky application out of source control and try to find out what actually caused the leak. Obviously wrong reasoning brought me to the right answer back then.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverbSetting some references to null inside an object you will no longer need isn't wrong of course; and if you somehow are keeping that object alive inadvertently, it would reduce the damage. That is probably why it happened to solve your problem. :)
Luc Pattyn [Forum Guidelines] [My Articles] Nil Volentibus Arduum
Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at the code.
-
Setting some references to null inside an object you will no longer need isn't wrong of course; and if you somehow are keeping that object alive inadvertently, it would reduce the damage. That is probably why it happened to solve your problem. :)
Luc Pattyn [Forum Guidelines] [My Articles] Nil Volentibus Arduum
Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at the code.
One application was simply requesting memory faster than the GC was releasing it. According to ProcessExplorer a thorough implementation of Dispose() and clearing all references corrected this. The other one, which I want to look at again tomorrow, is far trickier. Someone crated controls on an ASP .Net page dynamically and stored them in the session state in order to easily restore them after postbacks. Bad idea. Memory consumption went up with every page request. Ok, the controls, the SessionState object and the page itself formed loops. But where was the anchor that kept them alive? After finishing its lifecycle, the page should be released by the server and the whole construct should be floating. Edit: Reading my own post, the prime suspect right now would be the SessionState object. The server might actually keep it until the session expires and reuse it at every new request.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverb -
One application was simply requesting memory faster than the GC was releasing it. According to ProcessExplorer a thorough implementation of Dispose() and clearing all references corrected this. The other one, which I want to look at again tomorrow, is far trickier. Someone crated controls on an ASP .Net page dynamically and stored them in the session state in order to easily restore them after postbacks. Bad idea. Memory consumption went up with every page request. Ok, the controls, the SessionState object and the page itself formed loops. But where was the anchor that kept them alive? After finishing its lifecycle, the page should be released by the server and the whole construct should be floating. Edit: Reading my own post, the prime suspect right now would be the SessionState object. The server might actually keep it until the session expires and reuse it at every new request.
"I have what could be described as the most wide-open sense of humor on the site, and if I don't think something is funny, then it really isn't." - JSOC, 2011 -----
"Friar Modest never was a prior" - Italian proverbCDP1802 wrote:
requesting memory faster than the GC was releasing it
without finalizers, I would not expect that as most of the GC runs on the thread that needs the memory; finalizers however get handled by a separate (and low-priority) thread, which indeed can be an issue.
CDP1802 wrote:
the controls, the SessionState object and the page itself formed loops
I don't know how that got implemented exactly, and yes I would assume the SessionState is kept in say a static collection (a Dictionary?) so it persists until it has been decided the session is over. :)
Luc Pattyn [Forum Guidelines] [My Articles] Nil Volentibus Arduum
Please use <PRE> tags for code snippets, they preserve indentation, improve readability, and make me actually look at the code.