Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor
  • Default (No Skin)
  • No Skin
Collapse
Code Project
CODE PROJECT For Those Who Code
  • Home
  • Articles
  • FAQ
Community
  1. Home
  2. General Programming
  3. C#
  4. out of memory exception

out of memory exception

Scheduled Pinned Locked Moved C#
databaseperformance
26 Posts 6 Posters 0 Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M Mogaambo

    Paul Conrad wrote:

    Didn't you already ask this question once?

    Yes I did , but check yourself what answers i received, are they fruitful no no no no So this time I asked my question in a different manner. If, you can provide a good path then take thanks in Advance.

    “You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

    D Offline
    D Offline
    Dave Kreskowiak
    wrote on last edited by
    #21

    Mogaambo wrote:

    Yes I did , but check yourself what answers i received, are they fruitful no no no no

    Uhhh, yes they are. And you were told exactly why your approach wouldn't work. You simply ARE running the machine out of memory trying to read and retain all that data at once. You have no choice but to either read and process the data in small chunks, or farm the entire job out to SQL Server to do it for you.

    A guide to posting questions on CodeProject[^]
    Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
         2006, 2007, 2008

    1 Reply Last reply
    0
    • M Mogaambo

      Well first of all thanks for your valuable suggestions but as you are MVP and you say

      Colin Angus Mackay wrote:

      I give up.

      But Something for you. Sam Ewing: It's not the hours you put in your work that count, it's work you put in the hours. F. Scott Fitzgerald: You don't write because you want to say something; you write because you've got something to say.

      “You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

      D Offline
      D Offline
      Dave Kreskowiak
      wrote on last edited by
      #22

      Mogaambo wrote:

      Well first of all thanks for your valuable suggestions but as you are MVP and you say Colin Angus Mackay wrote: I give up. But Something for you.

      How about this. He's good at what he does because he recognizes that to keep working on a futile solution is a pointless.

      A guide to posting questions on CodeProject[^]
      Dave Kreskowiak Microsoft MVP Visual Developer - Visual Basic
           2006, 2007, 2008

      1 Reply Last reply
      0
      • realJSOPR realJSOP

        You're arguing with the guy that gave you a solution. Who here is the asshole?

        "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
        -----
        "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001

        M Offline
        M Offline
        Mogaambo
        wrote on last edited by
        #23

        Tell me where do i argued, i say thanks to his valuable suggestions and is my questions are pointless , no not they are. if he doesn't say anything to me then y your azz is itching.

        “You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

        1 Reply Last reply
        0
        • M Mogaambo

          Colin Angus Mackay wrote:

          Your strategy is to dump the entire contents of the files in to memory then write them to the database. Why?

          Because I am Passing That List Datables to a function whose purpose is to write data to table using sqlbulkcopy

          Colin Angus Mackay wrote:

          Why not just read the files one-by-one and write them to the database?

          So you want me to change strategy. 1. for( int i=0; i<totalnumberoffilestoscan;> { Step1: // Create A New DataTable Step2: // Call a Function That accepts the ith File and newly created DataTable as parameter and do the work of adding required number of Columns with their DataType and size Step3: // Call a function that will read the text file row by row and add them to datatable Step 4: // Pass this DataTable to another function whose purpose is to use SQlBulkCopy to write Records from the datatable. } Is This Strategy Feasible, will not be again give me out of memory Exception. in each loop i am creating a new Datatable ??????:confused::confused::confused:

          “You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

          A Offline
          A Offline
          Abi Bellamkonda
          wrote on last edited by
          #24

          25 files x 1 Million rows = 25 Million If you say each row is 1024 bytes. Its totally 25 GB of memory required. As you might already know, when RAM is not sufficent the system writes to harddisk & its slow. Unless you have free space of 40GB (to accomodate calculation issues), you cannot run your application & still it will be slow. So your approach is to break it up into pieces. You got few approaches: 1. Dont use datatables and use memory streams, use SqlCommand - this is more code, i guess 2. If you want to use datatables. You have to run the BulkCopy, for each datatable & call DataTable.Dispose() method (and may be call to Garbage collector, as Dispose will not free memory immediatly) after each use. 3. Use SQL Server, to import the files directly. DTS / SISS supports importing of various file formats, excel, csv, access. It will need a SQL command, which will take the file location & it will import it. You can google up for the sql syntax or use DTS Export / Import wizard & in last step, save the command.

          Abi ( Abishek Bellamkonda ) My Blog: http://abibaby.blogspot.com =(:*

          1 Reply Last reply
          0
          • C Colin Angus Mackay

            You are not paying attention. Or you are not bothering to understand transactions.

            Begin Transaction
            Start File Loop
            Start Data Chunk Loop
            Read upto 10,000 records from file
            Write those records to the database
            End Data Chunk Loop
            End File Loop
            End Transaction

            Each iteration around the Data Chunk Loop has one communication with the database (One "shot" per iteration) If you have a million records you have 100 "shots" at writing to the database.

            Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog

            M Offline
            M Offline
            Mogaambo
            wrote on last edited by
            #25

            Thanks alot. I have implemented your algorithm and its working fine and more fast then previous one. Millions of bows to you :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D and sorry if you hurt from any of mine statements.

            “You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

            modified on Tuesday, October 14, 2008 11:41 AM

            C 1 Reply Last reply
            0
            • M Mogaambo

              Thanks alot. I have implemented your algorithm and its working fine and more fast then previous one. Millions of bows to you :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D :-D and sorry if you hurt from any of mine statements.

              “You will never be a leader unless you first learn to follow and be led.” –Tiorio "Coming together is a beginning, staying together is progress, and working together is success." Henry Ford

              modified on Tuesday, October 14, 2008 11:41 AM

              C Offline
              C Offline
              Colin Angus Mackay
              wrote on last edited by
              #26

              Mogaambo wrote:

              more fast then previous one

              Probably because you are not thrashing the virtual memory's page file.

              Recent blog posts: *Method hiding Vs. overriding *Microsoft Surface *SQL Server / Visual Studio install order My Blog

              1 Reply Last reply
              0
              Reply
              • Reply as topic
              Log in to reply
              • Oldest to Newest
              • Newest to Oldest
              • Most Votes

              • Login
              • Don't have an account? Register
              • Login or register to search.
              • First post
                Last post
              0
              • Categories
              • Recent
              • Tags
              • Popular
              • World
              • Users
              • Groups