Memory leak trouble with Linq to Sql and multiple threads

JD86

I'm having some issue with my code in my Windows Service not releasing memory. In short my Windows Service basically created about 8 timers that elapsed every X minutes which called a method that queried data from Exchange and stored in the database. What I have noticed is the memory always climbed to the point it would crash the service when it reached maximum (32-bit process) I switched from using timers to using the Quartz library but experienced the same results. Here is an example of one of the timers jobs:

public class Get_MailboxDatabaseSizesTask : IJob
{
private static readonly ILog logger = LogManager.GetLogger("Get_MailboxDatabaseSizesTask");

    public void Execute(IJobExecutionContext context)
    {
        CloudPanelDbContext db = null;
        ExchActions powershell = null;
        int processedCount = 0;

        try
        {
            db = new CloudPanelDbContext(Config.ServiceSettings.SqlConnectionString);
            db.Connection.Open();

            powershell = new ExchActions();
            var mailboxDatabases = powershell.Get\_MailboxDatabaseSizes();
            db.StatMailboxDatabaseSizes.InsertAllOnSubmit(mailboxDatabases);
            db.SubmitChanges();

            processedCount = mailboxDatabases.Count;
            mailboxDatabases = null;
        }
        catch (Exception ex)
        {
            logger.ErrorFormat("Failed to retrieve mailbox database sizes: {0}", ex.ToString());
        }
        finally
        {
            if (powershell != null)
                powershell.Dispose();

            if (db != null)
                db.Dispose();

            logger.InfoFormat("Processed a total of {0} mailbox databases", processedCount);
        }
        
    }
}

As you can see I am disposing of my context and my ExchActions class... I solved the memory issue by adding this to the end of each Execute method:

        GC.Collect();
        GC.WaitForPendingFinalizers();

Its not ideal but its working.... but why do I have such a memory problem without it? Where should I look?

Dave Kreskowiak

Take out the GC crap. It's really not helping that much and, by the description you gave, will actually hide the problem, not fix it. From what you've said, it sounds like there's some other object you have to Dispose, but haven't, before it goes out of scope or you're constantly adding to a collection. There really isn't that much code here. I don't know where that logger is going to but it may also be the problem. Get a memory profiler[^]. It'll tell you what type the objects are that are taking up all that memory and give you a clue as to what to look for.

A guide to posting questions on CodeProject

Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak

Luc Pattyn

Hmm. Applying explicit GC operations is not supposed to make much of a difference. And actually context is the one item you are not disposing, and probably the caller of Execute() should take care of that, not Execute() itself. BTW: rather than finally, I prefer the using pattern for disposable objetcs; example:

using (ExchActions powershell = new ExchActions()) {
...
}

No matter how the using block is exited (normally, return, exception) the newly created object will always be disposed of correctly; and you can nest them as much as needed. I see you open a DB connection but never close it. There too you should be able to apply a using construct. DB connections get pooled, i.e. recycled, very well assuming you close or dispose them in time. Otherwise the system may exhaust the default pool and try to create large numbers of connections... :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Dave Kreskowiak

Luc has returned! :) Where the hell have you been for the last, oh, 3 years? Nothing bad I hope!

A guide to posting questions on CodeProject

Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak

Luc Pattyn

Hi Dave, I switched to a more passive mode a couple of years ago; I'm still reading a few CP articles each week, but I stopped my forum activities, lacking a sufficient number of decent questions since mediocrity introduced by QA started infecting the forums too. However, when I see something worthwhile in C# or algo forums, I occasionally post a bit. So, no, not really a return, just passing by. :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Dave Kreskowiak

Luc Pattyn wrote:

lacking a sufficient number of decent questions since mediocrity introduced by QA started infecting the forums too.

I hear that! The number of "questions" where the OP doesn't know what a computer is let alone how to write code for one just keeps growing and growing. The Copy'Paste'n'PrayItWorks coders are outnumbering the good questions by a large margin now-a-days. Too many people find it easier to ask someone to do their work for them or fix their code for them and won't even bother to set a breakpoint let alone look at variables in the debugger. It's really sad. It's wearing pretty thin on me too. I weep for the future of software development.

A guide to posting questions on CodeProject

Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak

Lost User

I've used Entity Framework and never yet found it necessary to use:

db.Connection.Open();

Sounds like this reference may apply to you: Entity Framework Connection Management[^]

Quote:

Behavior in EF6 and future versions For EF6 and future versions we have taken the approach that if the calling code chooses to open the connection by calling context.Database.Connection.Open() then it has a good reason for doing so and the framework will assume that it wants control over opening and closing of the connection and will no longer close the connection automatically. Note: This can potentially lead to connections which are open for a long time so use with care.

Dave Kreskowiak

I have once. I used EF to execute a direct SQL statement to start a job.

A guide to posting questions on CodeProject

Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak

Lost User

That sounds like a good reason.

JD86

I'm disposing of "db" which should close the connection shouldn't it? I can try using but I usually steer away from it because I can't really catch exceptions without putting another try/catch inside the finally which is in effect making it a nested try/catch isn't it?

JD86

Anytime I'm doing multiple queries I manually open the connection. From my understanding each query (without manually opening) is in essence doing this: -> Open Connection -> Query -> Close connection -> Open Connection -> Query -> Close connection So by me calling: db.Database.Connection.Open() it should be doing this: -> Open Connection -> Query -> Query -> Close connection

JD86

I will look into the memory profiler. The powershell commands i'm using is the same code I'm using in a web application and haven't noticed any problems. I thought it was the timers but I think its more LINQ to SQL since I swapped out the timers for the Quartz library. The GC.Collect didn't actually help because when it ran I could see the memory go up and go down... however after a couple days i'm seeing i'm at 1.5GB now. So it is possible that 1 timer is causing all this.... the logging is log4net and it logs to a text file

Luc Pattyn

Quote:

I'm disposing of "db" which should close the connection shouldn't it?

I don't know as I'm not familiar with CloudPanelDbContext. And I wouldn't rely on it unless (a) it is stated explicitly in trustworthy documentation and (b) I put in a comment repeating such fact.

Quote:

I can try using but ...

Yes you can nest try/catch/finally blocks, however it doesn't look good and most often there is no need to do so. Actually you often don't need an explicit finally block if all disposable objects got created with using constructs, that is the beauty of it: it is hard to get it wrong, as the code is much simpler (the compiler automatically translates it to something similar to what you have, but readability is much better, so error probability goes down). Your method, without corrections for the possible connection issues, would be simplified to:

public void Execute(IJobExecutionContext context) {
int processedCount = 0;
try {
using (CloudPanelDbContext db = new CloudPanelDbContext (Config.ServiceSettings.SqlConnectionString)) {
db.Connection.Open();
using (ExchActions powershell = new ExchActions()) {
var mailboxDatabases = powershell.Get_MailboxDatabaseSizes();
db.StatMailboxDatabaseSizes.InsertAllOnSubmit(mailboxDatabases);
db.SubmitChanges();
processedCount = mailboxDatabases.Count;
mailboxDatabases = null;
}
}
} catch (Exception exc) {
logger.ErrorFormat("Failed to ...");
}
logger.InfoFormat("Processed a total of {0} mailbox databases", processedCount);
}

I'm also not familiar with Entity Framework, however assuming your CloudPanelDbContext is derived from System.Data.Entity.DbContext a quick google session told me such context knows how to deal with open connections; it seems you can open a connection and keep it associated with your context, see e.g. some of the constructors here[^]. May I suggest you read up on that topic. PS: the same webpage also describes Dispose() with "The connection to the database

JD86

ClousePanelDbContext is actually from DataContext since this is Linq to SQL and not actually Entity Framework. (System.Data.Linq.DataContext) As far as the using statement I see you are putting that in the try/catch so if what is INSIDE the using statement throws an exception will it be caught in the catch area so I can see what the exception was? It was my understand that the using statement wouldn't throw the error where you could catch it unless you put the try/catch INSIDE the using statement. But then again I could of understood that wrong.

Dave Kreskowiak

Could it be the timer itself? Nope. But, it will most likely be in the code the timer executes.

A guide to posting questions on CodeProject

Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak

Lost User

Here is the remarks section for your "Connection" property:

Quote:

You can use this property to interoperate with relational ADO.NET code. The returned connection will be closed unless it has been explicitly opened by the user.

(Which appears to also reflect what is in the EF docs). DataContext.Connection Property (System.Data.Linq)[^]

Lost User

Hey Luc, welcome back.

Luc Pattyn

Hi Richard, this is just a sporadic visit, nothing permanent. :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

Luc Pattyn

1. Yes, a try/catch/finally construct guards everything the thread does inside the try block no matter how many using blocks, method calls, etc it performs (up to a nested try/catch f course). If any using constructs appear, the compiler transforms them to its own set of try/finally blocks (there is no USING instruction in IL!) which would then sit inside my try block; so any exception will be caught by my catch block. 2. I found this[^] stating your connections are not disposed off at all by disposing the db object. :)

Luc Pattyn [My Articles] Nil Volentibus Arduum

JD86

Ok as a test I took out all of the following:

db.Connection.Open();

I kept my Try/Catch/Finally which calls Dispose on the DataContext for testing. Since I am not manually opening connections it should open and close after each query/update but it should handle the connections by itself. I'm going to keep an eye on memory today. If this doesn't work then I will try the USING statement. I still want to handle the connection because I don't want it opening and closing all the time since it causes a delay. Once I narrow down what it is I'll call Close() before disposing of the DataContext. I will let you all know once I have an update