out of memory exception

Mogaambo

With your suggestion reading from file and writing to database should be done in 1 shot. For eg. WriteToDataBase(string filename) { //Begin SqlTransaction Begin loop to read 10000 records at a time (While all the records are not read) //write in a datatable //End loop //End SqlTransaction } // Some thing like this you want me to do

“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

Colin Angus Mackay

You are not paying attention. Or you are not bothering to understand transactions.

Begin Transaction
Start File Loop
Start Data Chunk Loop
Read upto 10,000 records from file
Write those records to the database
End Data Chunk Loop
End File Loop
End Transaction

Each iteration around the Data Chunk Loop has one communication with the database (One "shot" per iteration) If you have a million records you have 100 "shots" at writing to the database.

Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog

Mogaambo

Thanks for your quick reply, But,

Colin Angus Mackay wrote:

If you have a million records you have 100 "shots" at writing to the database.

Is this strategy fast enough , because to write 25 files it would take approx. 25 * 100= 2500 shots Means 2500 iteration will be performed for writing 25 files.

“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

Colin Angus Mackay

Well, your choice is going slightly slower, use SSIS (as someone else suggested) or running out of memory - You choose! I give up.

Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog

Mogaambo

Well first of all thanks for your valuable suggestions but as you are MVP and you say

Colin Angus Mackay wrote:

I give up.

But Something for you. Sam Ewing: It's not the hours you put in your work that count, it's work you put in the hours. F. Scott Fitzgerald: You don't write because you want to say something; you write because you've got something to say.

“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

realJSOP

He gave up because you're an asshat. Be a programmer for god's sake, and implement a solution.

"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001

Mogaambo

John Simmons / outlaw programmer wrote:

asshat

Thank you very very much AssHole

“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

realJSOP

You're arguing with the guy that gave you a solution. Who here is the asshole?

"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001

Dave Kreskowiak

Mogaambo wrote:

Yes I did , but check yourself what answers i received, are they fruitful no no no no

Uhhh, yes they are. And you were told exactly why your approach wouldn't work. You simply ARE running the machine out of memory trying to read and retain all that data at once. You have no choice but to either read and process the data in small chunks, or farm the entire job out to SQL Server to do it for you.

A guide to posting questions on CodeProject[^]
Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
2006, 2007, 2008

Dave Kreskowiak

Mogaambo wrote:

Well first of all thanks for your valuable suggestions but as you are MVP and you say Colin Angus Mackay wrote: I give up. But Something for you.

How about this. He's good at what he does because he recognizes that to keep working on a futile solution is a pointless.

A guide to posting questions on CodeProject[^]
Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
2006, 2007, 2008

Mogaambo

Tell me where do i argued, i say thanks to his valuable suggestions and is my questions are pointless , no not they are. if he doesn't say anything to me then y your azz is itching.

“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

Abi Bellamkonda

25 files x 1 Million rows = 25 Million If you say each row is 1024 bytes. Its totally 25 GB of memory required. As you might already know, when RAM is not sufficent the system writes to harddisk & its slow. Unless you have free space of 40GB (to accomodate calculation issues), you cannot run your application & still it will be slow. So your approach is to break it up into pieces. You got few approaches: 1. Dont use datatables and use memory streams, use SqlCommand - this is more code, i guess 2. If you want to use datatables. You have to run the BulkCopy, for each datatable & call DataTable.Dispose() method (and may be call to Garbage collector, as Dispose will not free memory immediatly) after each use. 3. Use SQL Server, to import the files directly. DTS / SISS supports importing of various file formats, excel, csv, access. It will need a SQL command, which will take the file location & it will import it. You can google up for the sql syntax or use DTS Export / Import wizard & in last step, save the command.

Abi ( Abishek Bellamkonda ) My Blog: http://abibaby.blogspot.com =(:*

Mogaambo

Thanks alot. I have implemented your algorithm and its working fine and more fast then previous one. Millions of bows to you :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D and sorry if you hurt from any of mine statements.

“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

modified on Tuesday, October 14, 2008 11:41 AM

Colin Angus Mackay · modified on Tuesday, October 14, 2008 11:41 AM

Mogaambo wrote:

more fast then previous one

Probably because you are not thrashing the virtual memory's page file.

Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog