Garbage Collection needed?
-
I have created a class to hold some related data (ResultsOfTest). I am reading a record from an Access file, loading the data into the members of the ResultsOfTest class, then inserting into a database. This works properly, except I am consuming massive amounts of memory. My understanding was the memory would be released when the new'ed object went out of scope (which should be at the end of the while loop). This doesn't seem to be the case, so I stuck in a garbage collection collect call every 1000 iterations. This doesn't seem to have any effect on the memory consumption.
while (readerAccess.Read()) { ResultsOfTest testResult = new ResultsOfTest(); try { if (!readerAccess.IsDBNull(iTestIndex)) { testResult.TestID = Convert.ToDecimal(readerAccess.GetInt32(iTestIndex)); } if (!readerAccess.IsDBNull(iEmployeeIndex)) { testResult.EmployeeID = Convert.ToDecimal(readerAccess.GetInt32(iEmployeeIndex)); } if (!readerAccess.IsDBNull(iTestDateIndex)) { testResult.DateOfTest = readerAccess.GetDateTime(iTestDateIndex); } if (!readerAccess.IsDBNull(iReasonIndex)) { testResult.ReasonForTest = Convert.ToDecimal(readerAccess.GetInt32(iReasonIndex)); } if (!readerAccess.IsDBNull(iDeferredDateIndex)) { testResult.DeferredDate = readerAccess.GetDateTime(iDeferredDateIndex); } if (!readerAccess.IsDBNull(iReadDateIndex)) { testResult.ReadDate = readerAccess.GetDateTime(iReadDateIndex); } if (!readerAccess.IsDBNull(iResultIndex)) { testResult.TestResultCode = Convert.ToDecimal(readerAccess.GetInt32(iResultIndex)); } if (!readerAccess.IsDBNull(iCommentIndex)) { testResult.Comments = readerAccess.GetString(iCommentIndex); } iRecordsRead++; } catch (OleDbException odex) { logger.Error("Unable to read record from Access db. {0}", odex.ToString()); } if (testResult.TestID > 0) { if (db.UpdateTestResults(ref testResult)) { iRecordsWritten++; } } if ((iRecordsRead % 1000) == 0) { StatusScreen.SetStatus("Garbage collection in process..."); GC.Collect(); } }
What can/should I do differently, so the program will release the un-needed memory? Thanks, Glenn -- modified at 10:57 Wednesday 15th March, 2006 (placed pre inside the code marker) -
I have created a class to hold some related data (ResultsOfTest). I am reading a record from an Access file, loading the data into the members of the ResultsOfTest class, then inserting into a database. This works properly, except I am consuming massive amounts of memory. My understanding was the memory would be released when the new'ed object went out of scope (which should be at the end of the while loop). This doesn't seem to be the case, so I stuck in a garbage collection collect call every 1000 iterations. This doesn't seem to have any effect on the memory consumption.
while (readerAccess.Read()) { ResultsOfTest testResult = new ResultsOfTest(); try { if (!readerAccess.IsDBNull(iTestIndex)) { testResult.TestID = Convert.ToDecimal(readerAccess.GetInt32(iTestIndex)); } if (!readerAccess.IsDBNull(iEmployeeIndex)) { testResult.EmployeeID = Convert.ToDecimal(readerAccess.GetInt32(iEmployeeIndex)); } if (!readerAccess.IsDBNull(iTestDateIndex)) { testResult.DateOfTest = readerAccess.GetDateTime(iTestDateIndex); } if (!readerAccess.IsDBNull(iReasonIndex)) { testResult.ReasonForTest = Convert.ToDecimal(readerAccess.GetInt32(iReasonIndex)); } if (!readerAccess.IsDBNull(iDeferredDateIndex)) { testResult.DeferredDate = readerAccess.GetDateTime(iDeferredDateIndex); } if (!readerAccess.IsDBNull(iReadDateIndex)) { testResult.ReadDate = readerAccess.GetDateTime(iReadDateIndex); } if (!readerAccess.IsDBNull(iResultIndex)) { testResult.TestResultCode = Convert.ToDecimal(readerAccess.GetInt32(iResultIndex)); } if (!readerAccess.IsDBNull(iCommentIndex)) { testResult.Comments = readerAccess.GetString(iCommentIndex); } iRecordsRead++; } catch (OleDbException odex) { logger.Error("Unable to read record from Access db. {0}", odex.ToString()); } if (testResult.TestID > 0) { if (db.UpdateTestResults(ref testResult)) { iRecordsWritten++; } } if ((iRecordsRead % 1000) == 0) { StatusScreen.SetStatus("Garbage collection in process..."); GC.Collect(); } }
What can/should I do differently, so the program will release the un-needed memory? Thanks, Glenn -- modified at 10:57 Wednesday 15th March, 2006 (placed pre inside the code marker)Glenn E. Lanier II wrote:
My understanding was the memory would be released when the new'ed object went out of scope (which should be at the end of the while loop). This doesn't seem to be the case, so I stuck in a garbage collection collect call every 1000 iterations. This doesn't seem to have any effect on the memory consumption.
Garbage collection occurs when it needs to. When objects go out of scope they mearly become available for garbage collection. If you are looking at the memory in the Task Manager then it will be showing the amount it has reserved from the operating system, not the actual amount in use. There are a number of performance counters you can look at for .NET applications. They will give you a more accurate picture of what is going on. ColinMackay.net Scottish Developers are looking for speakers for user group sessions over the next few months. Do you want to know more?
-
Glenn E. Lanier II wrote:
My understanding was the memory would be released when the new'ed object went out of scope (which should be at the end of the while loop). This doesn't seem to be the case, so I stuck in a garbage collection collect call every 1000 iterations. This doesn't seem to have any effect on the memory consumption.
Garbage collection occurs when it needs to. When objects go out of scope they mearly become available for garbage collection. If you are looking at the memory in the Task Manager then it will be showing the amount it has reserved from the operating system, not the actual amount in use. There are a number of performance counters you can look at for .NET applications. They will give you a more accurate picture of what is going on. ColinMackay.net Scottish Developers are looking for speakers for user group sessions over the next few months. Do you want to know more?
Colin Angus Mackay wrote:
If you are looking at the memory in the Task Manager then it will be showing the amount it has reserved from the operating system, not the actual amount in use. There are a number of performance counters you can look at for .NET applications. They will give you a more accurate picture of what is going on.
Such as? I let this code run yesterday (without the GC.Collect()) on about 60000 records. I logged each update, and noticed that while I was getting > 1 insert/second initially, by record 20000 insert time was reduced (as was machine response time) and by the time I got to record 32000, I was getting an insert every two-three minutes. Task Manager showed this process was using ~485M of memory. As soon as I killed the process, memory usage (again, using task manager) dropped almost immediately. I started the import again (skipping the already imported records), and I saw similiar results. I'm open to any suggestion(s) that will allow this code to run efficiently. :confused: --G
-
I have created a class to hold some related data (ResultsOfTest). I am reading a record from an Access file, loading the data into the members of the ResultsOfTest class, then inserting into a database. This works properly, except I am consuming massive amounts of memory. My understanding was the memory would be released when the new'ed object went out of scope (which should be at the end of the while loop). This doesn't seem to be the case, so I stuck in a garbage collection collect call every 1000 iterations. This doesn't seem to have any effect on the memory consumption.
while (readerAccess.Read()) { ResultsOfTest testResult = new ResultsOfTest(); try { if (!readerAccess.IsDBNull(iTestIndex)) { testResult.TestID = Convert.ToDecimal(readerAccess.GetInt32(iTestIndex)); } if (!readerAccess.IsDBNull(iEmployeeIndex)) { testResult.EmployeeID = Convert.ToDecimal(readerAccess.GetInt32(iEmployeeIndex)); } if (!readerAccess.IsDBNull(iTestDateIndex)) { testResult.DateOfTest = readerAccess.GetDateTime(iTestDateIndex); } if (!readerAccess.IsDBNull(iReasonIndex)) { testResult.ReasonForTest = Convert.ToDecimal(readerAccess.GetInt32(iReasonIndex)); } if (!readerAccess.IsDBNull(iDeferredDateIndex)) { testResult.DeferredDate = readerAccess.GetDateTime(iDeferredDateIndex); } if (!readerAccess.IsDBNull(iReadDateIndex)) { testResult.ReadDate = readerAccess.GetDateTime(iReadDateIndex); } if (!readerAccess.IsDBNull(iResultIndex)) { testResult.TestResultCode = Convert.ToDecimal(readerAccess.GetInt32(iResultIndex)); } if (!readerAccess.IsDBNull(iCommentIndex)) { testResult.Comments = readerAccess.GetString(iCommentIndex); } iRecordsRead++; } catch (OleDbException odex) { logger.Error("Unable to read record from Access db. {0}", odex.ToString()); } if (testResult.TestID > 0) { if (db.UpdateTestResults(ref testResult)) { iRecordsWritten++; } } if ((iRecordsRead % 1000) == 0) { StatusScreen.SetStatus("Garbage collection in process..."); GC.Collect(); } }
What can/should I do differently, so the program will release the un-needed memory? Thanks, Glenn -- modified at 10:57 Wednesday 15th March, 2006 (placed pre inside the code marker)When you're done using the SqlDataReader, call Dispose() on it. This will release some unmanaged resources and may also allow some managed objects to eventually be freed. The same goes for your SqlCommand and your SqlConnection. Another thing might be your call to db.UpdateTestResults(ref testResult). If you're storing your testResult somewhere, it won't be freed, obviously. So perhaps your large consumption of memory is due to having lots of ResultsOfTest objects lying around. Go check out the CLR profiler[^], or use a tool like Ants Memory and Performance profiler[^].
Tech, life, family, faith: Give me a visit. I'm currently blogging about: Moral Muscle The apostle Paul, modernly speaking: Epistles of Paul Judah Himango
-
When you're done using the SqlDataReader, call Dispose() on it. This will release some unmanaged resources and may also allow some managed objects to eventually be freed. The same goes for your SqlCommand and your SqlConnection. Another thing might be your call to db.UpdateTestResults(ref testResult). If you're storing your testResult somewhere, it won't be freed, obviously. So perhaps your large consumption of memory is due to having lots of ResultsOfTest objects lying around. Go check out the CLR profiler[^], or use a tool like Ants Memory and Performance profiler[^].
Tech, life, family, faith: Give me a visit. I'm currently blogging about: Moral Muscle The apostle Paul, modernly speaking: Epistles of Paul Judah Himango
Judah Himango wrote:
When you're done using the SqlDataReader, call Dispose() on it. This will release some unmanaged resources and may also allow some managed objects to eventually be freed. The same goes for your SqlCommand and your SqlConnection.
I am disposing the OracleDataReader (and OracleCommand and OracleConnection). The only thing
Judah Himango wrote:
Another thing might be your call to db.UpdateTestResults(ref testResult). If you're storing your testResult somewhere, it won't be freed, obviously.
db.UpdateTestResults(ref testResult) simply builds the SQL (either insert or update), connects to the database, performs an ExecuteNonQuery, cleans up and returns. Thanks for the links -- I'll check them out. --G
-
When you're done using the SqlDataReader, call Dispose() on it. This will release some unmanaged resources and may also allow some managed objects to eventually be freed. The same goes for your SqlCommand and your SqlConnection. Another thing might be your call to db.UpdateTestResults(ref testResult). If you're storing your testResult somewhere, it won't be freed, obviously. So perhaps your large consumption of memory is due to having lots of ResultsOfTest objects lying around. Go check out the CLR profiler[^], or use a tool like Ants Memory and Performance profiler[^].
Tech, life, family, faith: Give me a visit. I'm currently blogging about: Moral Muscle The apostle Paul, modernly speaking: Epistles of Paul Judah Himango
Judah Himango wrote:
Go check out the CLR profiler
Working Link: http://www.microsoft.com/downloads/details.aspx?FamilyId=86CE6052-D7F4-4AEB-9B7A-94635BEEBDDA&displaylang=en [^]
-
Judah Himango wrote:
When you're done using the SqlDataReader, call Dispose() on it. This will release some unmanaged resources and may also allow some managed objects to eventually be freed. The same goes for your SqlCommand and your SqlConnection.
I am disposing the OracleDataReader (and OracleCommand and OracleConnection). The only thing
Judah Himango wrote:
Another thing might be your call to db.UpdateTestResults(ref testResult). If you're storing your testResult somewhere, it won't be freed, obviously.
db.UpdateTestResults(ref testResult) simply builds the SQL (either insert or update), connects to the database, performs an ExecuteNonQuery, cleans up and returns. Thanks for the links -- I'll check them out. --G
Ok, well if the reality of things is that you're allocating too many objects before the GC can collect them, you are certainly free to call GC.Collect(). It's not recommended as the GC supposedly adapts to its usage, but you're not the first case I've read where the GC has become too lazy.
Tech, life, family, faith: Give me a visit. I'm currently blogging about: Moral Muscle The apostle Paul, modernly speaking: Epistles of Paul Judah Himango