Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Code Project
CODE PROJECT For Those Who Code
  • Home
  • Articles
  • FAQ
Community
  1. Home
  2. Database & SysAdmin
  3. Database
  4. multiple threads [modified]

multiple threads [modified]

Scheduled Pinned Locked Moved Database
databasesqlitehelpquestion
12 Posts 5 Posters 0 Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • L loyal ginger

    It's hard to find any database engine that does not use .dll files. SQLite can be linked to your program so that a .dll file is not needed. However, you mentioned that it's not sutible for your project. You can use multiple threads to write to an SQL Server database, but the actual performance of the writing operation may not be better than single-threaded programs. The speed is limited by the performance of the hard drive and whether your database is setup to run in distributted environment. If your database resides on a single hard drive, I doubt that you will see any performance improvement by using multi-threading. You can do a lot more on high-end database products. The question is whether or not your boss is willing to spend a lot of money on high-end database products?

    J Offline
    J Offline
    Jack_18
    wrote on last edited by
    #3

    thanks for answering the reason that I don't want to use sqlite are: 1. multi-thread writing is not supported. 2. it requires a sqlite.dll file in the exe folder to work. If you think that performance wouldn't differ much using multiple threads then i can very well use a single thread. I'll carry out the tests. If you know a method so that my app can become a two file app ie one exe and one database file then please tell me. in the above post ANDY_L_J has pointed out that .sdf databases need 7 dlls. if that's that then using sqlite is a better option as it requires only 1 dll file.

    L 1 Reply Last reply
    0
    • J Jack_18

      Hi all I am using VB 2008. I am developing a program that writes millions of lines to a database. The type of database isn't decided yet. I think I'll be using SQL databases (sdf) that VB provides. I need to add millions of rows at one time. Can I do that by using multiple threads? I mean I'll divide the data in ,say, 6 parts and write each part using a different thread. My whole purpose is to save time. As fast as the writing occurs my boss is that happier. He doesn't want to know how that is being done, he just wants to know that its been done. I know sqlite can't do this. Is there any other database which is able to do that. It must not use any dll. My boss wants the app to be independent. Would that result in data loss? even if the rows are written out of order, no problem. -- Modified Sunday, May 16, 2010 7:27 AM

      L Offline
      L Offline
      Luc Pattyn
      wrote on last edited by
      #4

      Do you need a database at all? is your app going to have multiple users, sharing the data? If you don't need the existing data to generate new data, you could (each) create a simple file, then perform a "bulk insert". :)

      Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]


      I only read formatted code with indentation, so please use PRE tags for code snippets.


      I'm not participating in frackin' Q&A, so if you want my opinion, ask away in a real forum (or on my profile page).


      J 1 Reply Last reply
      0
      • L Luc Pattyn

        Do you need a database at all? is your app going to have multiple users, sharing the data? If you don't need the existing data to generate new data, you could (each) create a simple file, then perform a "bulk insert". :)

        Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]


        I only read formatted code with indentation, so please use PRE tags for code snippets.


        I'm not participating in frackin' Q&A, so if you want my opinion, ask away in a real forum (or on my profile page).


        J Offline
        J Offline
        Jack_18
        wrote on last edited by
        #5

        Hullo Luc I need a database as I need to perform some operations that only a database can do effectively. no its not a multiple user program. I have a large quantity of data that I want to write to the database as fast as possible. I thought of dividing the data into ,say, 5 parts and use 5 threads to put these 5 parts so that my work is completed in 1/5th time. Am I correct in thinking so? If I do that would that result in data loss?

        L L 2 Replies Last reply
        0
        • J Jack_18

          Hullo Luc I need a database as I need to perform some operations that only a database can do effectively. no its not a multiple user program. I have a large quantity of data that I want to write to the database as fast as possible. I thought of dividing the data into ,say, 5 parts and use 5 threads to put these 5 parts so that my work is completed in 1/5th time. Am I correct in thinking so? If I do that would that result in data loss?

          L Offline
          L Offline
          Luc Pattyn
          wrote on last edited by
          #6

          If all that happens on a single machine, I doubt having multiple threads will really help, but I haven't tried it. I'd rather go for fewer SQL commands, so I am in favor of bulk insert, if that fits the application. :)

          Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]


          I only read formatted code with indentation, so please use PRE tags for code snippets.


          I'm not participating in frackin' Q&A, so if you want my opinion, ask away in a real forum (or on my profile page).


          1 Reply Last reply
          0
          • J Jack_18

            Hullo Luc I need a database as I need to perform some operations that only a database can do effectively. no its not a multiple user program. I have a large quantity of data that I want to write to the database as fast as possible. I thought of dividing the data into ,say, 5 parts and use 5 threads to put these 5 parts so that my work is completed in 1/5th time. Am I correct in thinking so? If I do that would that result in data loss?

            L Offline
            L Offline
            Lost User
            wrote on last edited by
            #7

            Jack_18 wrote:

            I have a large quantity of data that I want to write to the databaseas fast as possible.

            A database isn't optimized for bulk logging in realtime. If "as fast as possible" takes five seconds, would that be considered acceptable?

            Jack_18 wrote:

            I thought of dividing the data into ,say, 5 parts and use 5 threads to put these 5 partsso that my work is completed in 1/5th time.

            Adding a thread doesn't mean that your work is done in half the time. How many threads can your system run simultaneously? If you need to log a lot of data, consider dumping it as suggested and have a second proces write it to a database.

            I are Troll :suss:

            1 Reply Last reply
            0
            • J Jack_18

              thanks for answering the reason that I don't want to use sqlite are: 1. multi-thread writing is not supported. 2. it requires a sqlite.dll file in the exe folder to work. If you think that performance wouldn't differ much using multiple threads then i can very well use a single thread. I'll carry out the tests. If you know a method so that my app can become a two file app ie one exe and one database file then please tell me. in the above post ANDY_L_J has pointed out that .sdf databases need 7 dlls. if that's that then using sqlite is a better option as it requires only 1 dll file.

              L Offline
              L Offline
              loyal ginger
              wrote on last edited by
              #8

              I am afraid I don't have a suggestion to your case. Since you are using VB, the need for dll seems to be unavoidable. Note that in the case of SQLite, C/C++ programs can link the whole SQLite into the program so you end up with just one exe file with no need for any dll files. That meets your "no dll" requirement but it's not using VB. Too bad.

              1 Reply Last reply
              0
              • J Jack_18

                Hi all I am using VB 2008. I am developing a program that writes millions of lines to a database. The type of database isn't decided yet. I think I'll be using SQL databases (sdf) that VB provides. I need to add millions of rows at one time. Can I do that by using multiple threads? I mean I'll divide the data in ,say, 6 parts and write each part using a different thread. My whole purpose is to save time. As fast as the writing occurs my boss is that happier. He doesn't want to know how that is being done, he just wants to know that its been done. I know sqlite can't do this. Is there any other database which is able to do that. It must not use any dll. My boss wants the app to be independent. Would that result in data loss? even if the rows are written out of order, no problem. -- Modified Sunday, May 16, 2010 7:27 AM

                H Offline
                H Offline
                Henry Minute
                wrote on last edited by
                #9

                Have a Google on '.net embedded database'. The first link offered has several suggestions and the fifth (judging purely from the subject line) might fit your needs. Good luck! :)

                Henry Minute Do not read medical books! You could die of a misprint. - Mark Twain Girl: (staring) "Why do you need an icy cucumber?" “I want to report a fraud. The government is lying to us all.” Why do programmers often confuse Halloween and Christmas? Because 31 Oct = 25 Dec.

                L 1 Reply Last reply
                0
                • H Henry Minute

                  Have a Google on '.net embedded database'. The first link offered has several suggestions and the fifth (judging purely from the subject line) might fit your needs. Good luck! :)

                  Henry Minute Do not read medical books! You could die of a misprint. - Mark Twain Girl: (staring) "Why do you need an icy cucumber?" “I want to report a fraud. The government is lying to us all.” Why do programmers often confuse Halloween and Christmas? Because 31 Oct = 25 Dec.

                  L Offline
                  L Offline
                  Luc Pattyn
                  wrote on last edited by
                  #10

                  Hi Henry, not sure referring to Google results by their position on the screen works well for different countries. By default, in Belgium we get a localized Google site for starters; then there may be payed results on top; and of course, items move around when they become more/less popular. This is what I am currently getting for your suggested Google action (using www.google.com, not .be):

                    Raima Embedded Databases
                    eXtremeDB Embedded DBMS
                    Embedded Database
                    We need an Embedded Database for .NET applications - Jon Galloway
                    SQLite - ALMOST a great embedded database solution for .NET ...
                    Embedded SQL database engine for .NET developers - VistaDB
                    Embedded .Net database shootout
                    DotNetFirebird: Using Firebird SQL in .NET: Embedded Firebird and 
                    ...
                  

                  do you see the same list? :)

                  Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]


                  I only read formatted code with indentation, so please use PRE tags for code snippets.


                  I'm not participating in frackin' Q&A, so if you want my opinion, ask away in a real forum (or on my profile page).


                  H 1 Reply Last reply
                  0
                  • L Luc Pattyn

                    Hi Henry, not sure referring to Google results by their position on the screen works well for different countries. By default, in Belgium we get a localized Google site for starters; then there may be payed results on top; and of course, items move around when they become more/less popular. This is what I am currently getting for your suggested Google action (using www.google.com, not .be):

                      Raima Embedded Databases
                      eXtremeDB Embedded DBMS
                      Embedded Database
                      We need an Embedded Database for .NET applications - Jon Galloway
                      SQLite - ALMOST a great embedded database solution for .NET ...
                      Embedded SQL database engine for .NET developers - VistaDB
                      Embedded .Net database shootout
                      DotNetFirebird: Using Firebird SQL in .NET: Embedded Firebird and 
                      ...
                    

                    do you see the same list? :)

                    Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]


                    I only read formatted code with indentation, so please use PRE tags for code snippets.


                    I'm not participating in frackin' Q&A, so if you want my opinion, ask away in a real forum (or on my profile page).


                    H Offline
                    H Offline
                    Henry Minute
                    wrote on last edited by
                    #11

                    Luc Pattyn wrote:

                    not sure referring to Google results by their position on the screen works well for different countries

                    Er........ :doh: I should have known that. As far as the list goes mine (google.co.uk) starts with the 4th item in your list and then follows exactly. The first two do not appear on the first page but the 3rd is. Still, if the OP did search he should have got some ideas at the very least.

                    Henry Minute Do not read medical books! You could die of a misprint. - Mark Twain Girl: (staring) "Why do you need an icy cucumber?" “I want to report a fraud. The government is lying to us all.” Why do programmers often confuse Halloween and Christmas? Because 31 Oct = 25 Dec.

                    L 1 Reply Last reply
                    0
                    • H Henry Minute

                      Luc Pattyn wrote:

                      not sure referring to Google results by their position on the screen works well for different countries

                      Er........ :doh: I should have known that. As far as the list goes mine (google.co.uk) starts with the 4th item in your list and then follows exactly. The first two do not appear on the first page but the 3rd is. Still, if the OP did search he should have got some ideas at the very least.

                      Henry Minute Do not read medical books! You could die of a misprint. - Mark Twain Girl: (staring) "Why do you need an icy cucumber?" “I want to report a fraud. The government is lying to us all.” Why do programmers often confuse Halloween and Christmas? Because 31 Oct = 25 Dec.

                      L Offline
                      L Offline
                      Luc Pattyn
                      wrote on last edited by
                      #12

                      yep, Google is American, and very NTSC at that, Never The Same Content. As is most of the internet. :laugh:

                      Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]


                      I only read formatted code with indentation, so please use PRE tags for code snippets.


                      I'm not participating in frackin' Q&A, so if you want my opinion, ask away in a real forum (or on my profile page).


                      1 Reply Last reply
                      0
                      Reply
                      • Reply as topic
                      Log in to reply
                      • Oldest to Newest
                      • Newest to Oldest
                      • Most Votes


                      • Login

                      • Don't have an account? Register

                      • Login or register to search.
                      • First post
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • World
                      • Users
                      • Groups