best local database for large tables? (C#, .net)
-
I just got handed a challange: Create a program to knead stellar data into something visual. I know this is redundant, as the tools already exist, but I'm between jobs and need something to do. So I have access to the data, which is a HUGE csv file. said CSV dataset keeps crashing, excel and access, so I need some local, "portable"/deployable and free database technology. Since I want some scalability, and decleared datatypes, using a CSV file isn't feasable. My first thought is a sql server, but they're not real user friendly if I want to distribute the software. Anyone have any suggestions? Please explain the benefits of the technology and not just name thibgs Thanks!
How big is this file? Both in bytes and in rows.
Wrong is evil and must be defeated. - Jeff Ello
-
How big is this file? Both in bytes and in rows.
Wrong is evil and must be defeated. - Jeff Ello
60MB and 300'000++ rows
-
60MB and 300'000++ rows
-
60MB and 300'000++ rows
(Almost) Any database can handle that. Take a look at SQLite[^] for minimal installation and configuration. Or SQL Server Compact edition as already mentioned, if you want to be able to upgrade to a larger version later on. There are several inbetween versions of SQL-Server, there's an ok overview here[^]
Wrong is evil and must be defeated. - Jeff Ello
-
I just got handed a challange: Create a program to knead stellar data into something visual. I know this is redundant, as the tools already exist, but I'm between jobs and need something to do. So I have access to the data, which is a HUGE csv file. said CSV dataset keeps crashing, excel and access, so I need some local, "portable"/deployable and free database technology. Since I want some scalability, and decleared datatypes, using a CSV file isn't feasable. My first thought is a sql server, but they're not real user friendly if I want to distribute the software. Anyone have any suggestions? Please explain the benefits of the technology and not just name thibgs Thanks!
A lot depends on what you are going to do with the data. If you are going to be reading, inserting, updating and deleting then a database is the way to go - SQLite is a good choice. If all you need to do is read a large amount of data into an application and never need to update this data then you could consider Serialising your classes(or collections of classes). This is what I have done for a calorie counting application and a visual screening application that I wrote(even if you are applying updates serialisation is still worth considering). The nice bit about serialisation into XML is that you can look at your data with something like Notepad++ and make changes manually where necessary.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
-
How about SQL Compact Edition[^].
Apparently it's not the case (or not yet?) but I was under the impression that SQL Compact was deprecated. Did MS change their mind or am I confusing it with something "similar"?
If the brain were so simple we could understand it, we would be so simple we couldn't. — Lyall Watson
-
Apparently it's not the case (or not yet?) but I was under the impression that SQL Compact was deprecated. Did MS change their mind or am I confusing it with something "similar"?
If the brain were so simple we could understand it, we would be so simple we couldn't. — Lyall Watson
-
I just got handed a challange: Create a program to knead stellar data into something visual. I know this is redundant, as the tools already exist, but I'm between jobs and need something to do. So I have access to the data, which is a HUGE csv file. said CSV dataset keeps crashing, excel and access, so I need some local, "portable"/deployable and free database technology. Since I want some scalability, and decleared datatypes, using a CSV file isn't feasable. My first thought is a sql server, but they're not real user friendly if I want to distribute the software. Anyone have any suggestions? Please explain the benefits of the technology and not just name thibgs Thanks!
I suggest System.Data.SQLite: Downloads Page[^]. It is up to date, easy to use from c#, there is no service needed and it has very fast inserts and retrievals into/from tables. Last August I tested it by inserting up to 100 million of simple unique records on a Win 7 Pro with 4 GB of RAM and I could retrieve data in milliseconds, even from a table of 100 million... which I found quite impressive. When inserting I used parametrized queries. For fast retrieval an index needs to be created on a table column using SQL. Of course it depends on how complex is your data. I hope this helps...
-
I just got handed a challange: Create a program to knead stellar data into something visual. I know this is redundant, as the tools already exist, but I'm between jobs and need something to do. So I have access to the data, which is a HUGE csv file. said CSV dataset keeps crashing, excel and access, so I need some local, "portable"/deployable and free database technology. Since I want some scalability, and decleared datatypes, using a CSV file isn't feasable. My first thought is a sql server, but they're not real user friendly if I want to distribute the software. Anyone have any suggestions? Please explain the benefits of the technology and not just name thibgs Thanks!
SQL Server has a localdb version which seems to be the replacement for the compact edition.
What do you get when you cross a joke with a rhetorical question? The metaphorical solid rear-end expulsions have impacted the metaphorical motorized bladed rotating air movement mechanism. Do questions with multiple question marks annoy you???
-
A lot depends on what you are going to do with the data. If you are going to be reading, inserting, updating and deleting then a database is the way to go - SQLite is a good choice. If all you need to do is read a large amount of data into an application and never need to update this data then you could consider Serialising your classes(or collections of classes). This is what I have done for a calorie counting application and a visual screening application that I wrote(even if you are applying updates serialisation is still worth considering). The nice bit about serialisation into XML is that you can look at your data with something like Notepad++ and make changes manually where necessary.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
Ilove XML, but 300'000 rows, in a single file, times out the application
-
Ilove XML, but 300'000 rows, in a single file, times out the application
Sure :) 300,000 rows is probably going to stretch things a bit.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens