best local database for large tables? (C#, .net)
-
How big is this file? Both in bytes and in rows.
Wrong is evil and must be defeated. - Jeff Ello
60MB and 300'000++ rows
-
60MB and 300'000++ rows
-
60MB and 300'000++ rows
(Almost) Any database can handle that. Take a look at SQLite[^] for minimal installation and configuration. Or SQL Server Compact edition as already mentioned, if you want to be able to upgrade to a larger version later on. There are several inbetween versions of SQL-Server, there's an ok overview here[^]
Wrong is evil and must be defeated. - Jeff Ello
-
I just got handed a challange: Create a program to knead stellar data into something visual. I know this is redundant, as the tools already exist, but I'm between jobs and need something to do. So I have access to the data, which is a HUGE csv file. said CSV dataset keeps crashing, excel and access, so I need some local, "portable"/deployable and free database technology. Since I want some scalability, and decleared datatypes, using a CSV file isn't feasable. My first thought is a sql server, but they're not real user friendly if I want to distribute the software. Anyone have any suggestions? Please explain the benefits of the technology and not just name thibgs Thanks!
A lot depends on what you are going to do with the data. If you are going to be reading, inserting, updating and deleting then a database is the way to go - SQLite is a good choice. If all you need to do is read a large amount of data into an application and never need to update this data then you could consider Serialising your classes(or collections of classes). This is what I have done for a calorie counting application and a visual screening application that I wrote(even if you are applying updates serialisation is still worth considering). The nice bit about serialisation into XML is that you can look at your data with something like Notepad++ and make changes manually where necessary.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
-
How about SQL Compact Edition[^].
Apparently it's not the case (or not yet?) but I was under the impression that SQL Compact was deprecated. Did MS change their mind or am I confusing it with something "similar"?
If the brain were so simple we could understand it, we would be so simple we couldn't. — Lyall Watson
-
Apparently it's not the case (or not yet?) but I was under the impression that SQL Compact was deprecated. Did MS change their mind or am I confusing it with something "similar"?
If the brain were so simple we could understand it, we would be so simple we couldn't. — Lyall Watson
-
I just got handed a challange: Create a program to knead stellar data into something visual. I know this is redundant, as the tools already exist, but I'm between jobs and need something to do. So I have access to the data, which is a HUGE csv file. said CSV dataset keeps crashing, excel and access, so I need some local, "portable"/deployable and free database technology. Since I want some scalability, and decleared datatypes, using a CSV file isn't feasable. My first thought is a sql server, but they're not real user friendly if I want to distribute the software. Anyone have any suggestions? Please explain the benefits of the technology and not just name thibgs Thanks!
I suggest System.Data.SQLite: Downloads Page[^]. It is up to date, easy to use from c#, there is no service needed and it has very fast inserts and retrievals into/from tables. Last August I tested it by inserting up to 100 million of simple unique records on a Win 7 Pro with 4 GB of RAM and I could retrieve data in milliseconds, even from a table of 100 million... which I found quite impressive. When inserting I used parametrized queries. For fast retrieval an index needs to be created on a table column using SQL. Of course it depends on how complex is your data. I hope this helps...
-
I just got handed a challange: Create a program to knead stellar data into something visual. I know this is redundant, as the tools already exist, but I'm between jobs and need something to do. So I have access to the data, which is a HUGE csv file. said CSV dataset keeps crashing, excel and access, so I need some local, "portable"/deployable and free database technology. Since I want some scalability, and decleared datatypes, using a CSV file isn't feasable. My first thought is a sql server, but they're not real user friendly if I want to distribute the software. Anyone have any suggestions? Please explain the benefits of the technology and not just name thibgs Thanks!
SQL Server has a localdb version which seems to be the replacement for the compact edition.
What do you get when you cross a joke with a rhetorical question? The metaphorical solid rear-end expulsions have impacted the metaphorical motorized bladed rotating air movement mechanism. Do questions with multiple question marks annoy you???
-
A lot depends on what you are going to do with the data. If you are going to be reading, inserting, updating and deleting then a database is the way to go - SQLite is a good choice. If all you need to do is read a large amount of data into an application and never need to update this data then you could consider Serialising your classes(or collections of classes). This is what I have done for a calorie counting application and a visual screening application that I wrote(even if you are applying updates serialisation is still worth considering). The nice bit about serialisation into XML is that you can look at your data with something like Notepad++ and make changes manually where necessary.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
Ilove XML, but 300'000 rows, in a single file, times out the application
-
Ilove XML, but 300'000 rows, in a single file, times out the application
Sure :) 300,000 rows is probably going to stretch things a bit.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens