out of memory exception
-
Hi , I am getting out of memory exception, what i am trying to do is to read 25 text files, all files have over million records, and write them to database. My approach.... 1. Created a list of 25 datatable. 2. Read the column names from text file and add them to DataTable and with their datatype and size. 3. Pass this list to a function whose purpose is to read each file and add records to each of the datatable. It is just an rough but an important overview. In between i am running two progress bar. but i get out of memory exception. can anybody tell me any other approach......
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
Hi , I am getting out of memory exception, what i am trying to do is to read 25 text files, all files have over million records, and write them to database. My approach.... 1. Created a list of 25 datatable. 2. Read the column names from text file and add them to DataTable and with their datatype and size. 3. Pass this list to a function whose purpose is to read each file and add records to each of the datatable. It is just an rough but an important overview. In between i am running two progress bar. but i get out of memory exception. can anybody tell me any other approach......
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
Your strategy is to dump the entire contents of the files in to memory then write them to the database. Why? Why not just read the files one-by-one and write them to the database?
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
-
Hi , I am getting out of memory exception, what i am trying to do is to read 25 text files, all files have over million records, and write them to database. My approach.... 1. Created a list of 25 datatable. 2. Read the column names from text file and add them to DataTable and with their datatype and size. 3. Pass this list to a function whose purpose is to read each file and add records to each of the datatable. It is just an rough but an important overview. In between i am running two progress bar. but i get out of memory exception. can anybody tell me any other approach......
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
Didn't you already ask this question once? Actually, this is the 3rd time.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 -
Your strategy is to dump the entire contents of the files in to memory then write them to the database. Why? Why not just read the files one-by-one and write them to the database?
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
Colin Angus Mackay wrote:
Your strategy is to dump the entire contents of the files in to memory then write them to the database. Why?
Because I am Passing That List Datables to a function whose purpose is to write data to table using sqlbulkcopy
Colin Angus Mackay wrote:
Why not just read the files one-by-one and write them to the database?
So you want me to change strategy. 1. for( int i=0; i<totalnumberoffilestoscan;> { Step1: // Create A New DataTable Step2: // Call a Function That accepts the ith File and newly created DataTable as parameter and do the work of adding required number of Columns with their DataType and size Step3: // Call a function that will read the text file row by row and add them to datatable Step 4: // Pass this DataTable to another function whose purpose is to use SQlBulkCopy to write Records from the datatable. } Is This Strategy Feasible, will not be again give me out of memory Exception. in each loop i am creating a new Datatable ??????:confused::confused::confused:
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
Didn't you already ask this question once? Actually, this is the 3rd time.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001John Simmons / outlaw programmer wrote:
Didn't you already ask this question once?
Yes, he did, like 10 days ago and got some replies. Must be some amount of density involved.
"The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham
-
John Simmons / outlaw programmer wrote:
Didn't you already ask this question once?
Yes, he did, like 10 days ago and got some replies. Must be some amount of density involved.
"The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham
Paul Conrad wrote:
Didn't you already ask this question once?
Yes I did , but check yourself what answers i received, are they fruitful no no no no So this time I asked my question in a different manner. If, you can provide a good path then take thanks in Advance.
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
Paul Conrad wrote:
Didn't you already ask this question once?
Yes I did , but check yourself what answers i received, are they fruitful no no no no So this time I asked my question in a different manner. If, you can provide a good path then take thanks in Advance.
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
Mogaambo wrote:
check yourself what answers i received, are they fruitful
Yes, but this one and this other one are pretty right on the money.
"The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham
-
Mogaambo wrote:
check yourself what answers i received, are they fruitful
Yes, but this one and this other one are pretty right on the money.
"The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham
Paul Conrad wrote:
Yes, but this one and this other one are pretty right on the money.
I don't like their ideas because they say that you read the file 10000 lines and use the sqlBulkcopy to write that 10000 records and then again read 10000 records, but suppose if i read 10000 records from a file and successfully written them to table but on next time when i read next 10000 records and when in the time of writing records to table any exception occurs then my Table goes to inconsistent state, because 10000 records are in, but it should be like that either all data written or none of them should be written. Then...........................
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
Paul Conrad wrote:
Yes, but this one and this other one are pretty right on the money.
I don't like their ideas because they say that you read the file 10000 lines and use the sqlBulkcopy to write that 10000 records and then again read 10000 records, but suppose if i read 10000 records from a file and successfully written them to table but on next time when i read next 10000 records and when in the time of writing records to table any exception occurs then my Table goes to inconsistent state, because 10000 records are in, but it should be like that either all data written or none of them should be written. Then...........................
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
There's this really neat thing in databases called "transactions". Start a transaction, perform your operations, if all goes well you commit the transaction, if not then you rollback the transaction.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
-
Colin Angus Mackay wrote:
Your strategy is to dump the entire contents of the files in to memory then write them to the database. Why?
Because I am Passing That List Datables to a function whose purpose is to write data to table using sqlbulkcopy
Colin Angus Mackay wrote:
Why not just read the files one-by-one and write them to the database?
So you want me to change strategy. 1. for( int i=0; i<totalnumberoffilestoscan;> { Step1: // Create A New DataTable Step2: // Call a Function That accepts the ith File and newly created DataTable as parameter and do the work of adding required number of Columns with their DataType and size Step3: // Call a function that will read the text file row by row and add them to datatable Step 4: // Pass this DataTable to another function whose purpose is to use SQlBulkCopy to write Records from the datatable. } Is This Strategy Feasible, will not be again give me out of memory Exception. in each loop i am creating a new Datatable ??????:confused::confused::confused:
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
Mogaambo wrote:
Is This Strategy Feasible, will not be again give me out of memory Exception. in each loop i am creating a new Datatable ??????
I don't know if it will work for you because I don't know enough about your data. I do know that it will use a lot less memory than you are using now. If an individual file is large enough then you will still get an out of memory exception. It has been suggested to you else where that you read each file in chunks and process the chunks. That seems the most sensible strategy to me.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
-
There's this really neat thing in databases called "transactions". Start a transaction, perform your operations, if all goes well you commit the transaction, if not then you rollback the transaction.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
SqlTransaction will only be used for current 10000 records not for next 10000 records.. Like this step 1: //DataTable dt= ReadTenThousandLines(); Step 2: // Write_to_database (dt); In Write_to_database() function, I will use sqlBulkCopy nad their i Use SqlTransaction Class, but suppose if i again passing 10000 Record containing datatable and if it fails then ??????????????
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
SqlTransaction will only be used for current 10000 records not for next 10000 records.. Like this step 1: //DataTable dt= ReadTenThousandLines(); Step 2: // Write_to_database (dt); In Write_to_database() function, I will use sqlBulkCopy nad their i Use SqlTransaction Class, but suppose if i again passing 10000 Record containing datatable and if it fails then ??????????????
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
Mogaambo wrote:
SqlTransaction will only be used for current 10000 records not for next 10000 records..
That's a bit pointless.
Mogaambo wrote:
In Write_to_database() function, I will use sqlBulkCopy nad their i Use SqlTransaction Class, but suppose if i again passing 10000 Record containing datatable and if it fails then ??????????????
My advice has not changed. You use a transaction. You start it, you do all your bulk copying then you commit the transaction. You do NOT create one transaction for each batch of data. You create ONE transaction to cover everything.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
-
Mogaambo wrote:
SqlTransaction will only be used for current 10000 records not for next 10000 records..
That's a bit pointless.
Mogaambo wrote:
In Write_to_database() function, I will use sqlBulkCopy nad their i Use SqlTransaction Class, but suppose if i again passing 10000 Record containing datatable and if it fails then ??????????????
My advice has not changed. You use a transaction. You start it, you do all your bulk copying then you commit the transaction. You do NOT create one transaction for each batch of data. You create ONE transaction to cover everything.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
With your suggestion reading from file and writing to database should be done in 1 shot. For eg. WriteToDataBase(string filename) { //Begin SqlTransaction Begin loop to read 10000 records at a time (While all the records are not read) //write in a datatable //End loop //End SqlTransaction } // Some thing like this you want me to do
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
With your suggestion reading from file and writing to database should be done in 1 shot. For eg. WriteToDataBase(string filename) { //Begin SqlTransaction Begin loop to read 10000 records at a time (While all the records are not read) //write in a datatable //End loop //End SqlTransaction } // Some thing like this you want me to do
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
You are not paying attention. Or you are not bothering to understand transactions.
Begin Transaction
Start File Loop
Start Data Chunk Loop
Read upto 10,000 records from file
Write those records to the database
End Data Chunk Loop
End File Loop
End TransactionEach iteration around the Data Chunk Loop has one communication with the database (One "shot" per iteration) If you have a million records you have 100 "shots" at writing to the database.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
-
You are not paying attention. Or you are not bothering to understand transactions.
Begin Transaction
Start File Loop
Start Data Chunk Loop
Read upto 10,000 records from file
Write those records to the database
End Data Chunk Loop
End File Loop
End TransactionEach iteration around the Data Chunk Loop has one communication with the database (One "shot" per iteration) If you have a million records you have 100 "shots" at writing to the database.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
Thanks for your quick reply, But,
Colin Angus Mackay wrote:
If you have a million records you have 100 "shots" at writing to the database.
Is this strategy fast enough , because to write 25 files it would take approx. 25 * 100= 2500 shots Means 2500 iteration will be performed for writing 25 files.
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
Thanks for your quick reply, But,
Colin Angus Mackay wrote:
If you have a million records you have 100 "shots" at writing to the database.
Is this strategy fast enough , because to write 25 files it would take approx. 25 * 100= 2500 shots Means 2500 iteration will be performed for writing 25 files.
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
Well, your choice is going slightly slower, use SSIS (as someone else suggested) or running out of memory - You choose! I give up.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
-
Well, your choice is going slightly slower, use SSIS (as someone else suggested) or running out of memory - You choose! I give up.
Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog
Well first of all thanks for your valuable suggestions but as you are MVP and you say
Colin Angus Mackay wrote:
I give up.
But Something for you. Sam Ewing: It's not the hours you put in your work that count, it's work you put in the hours. F. Scott Fitzgerald: You don't write because you want to say something; you write because you've got something to say.
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
Well first of all thanks for your valuable suggestions but as you are MVP and you say
Colin Angus Mackay wrote:
I give up.
But Something for you. Sam Ewing: It's not the hours you put in your work that count, it's work you put in the hours. F. Scott Fitzgerald: You don't write because you want to say something; you write because you've got something to say.
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
He gave up because you're an asshat. Be a programmer for god's sake, and implement a solution.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 -
He gave up because you're an asshat. Be a programmer for god's sake, and implement a solution.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001John Simmons / outlaw programmer wrote:
asshat
Thank you very very much AssHole
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
-
John Simmons / outlaw programmer wrote:
asshat
Thank you very very much AssHole
“You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
You're arguing with the guy that gave you a solution. Who here is the asshole?
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001
