Simple question for the Database Rock Stars
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
There are no database rockstars, but those who master databases are a SELECT group.
-
There are no database rockstars, but those who master databases are a SELECT group.
ORDER, please!
No object is so beautiful that, under certain conditions, it will not look ugly. - Oscar Wilde
-
There are no database rockstars, but those who master databases are a SELECT group.
Duncan Edwards Jones wrote:
There are no database rockstars,
Well, there is "the One". Oracle told me so. Ok, I am out.
You have just been Sharapova'd.
-
There are no database rockstars, but those who master databases are a SELECT group.
Yeah, it's pretty hard to JOIN their RANKS! :D
Visit my blog at Sander's bits - Writing the code you need. Or read my articles at my CodeProject profile.
Simplicity is prerequisite for reliability. — Edsger W. Dijkstra
Regards, Sander
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
Databases do lots more than just store data. It's a centralized point for transactional data storage, modification and retrieval. Try using some XML files in an enterprise environment. That's simply not going to work. If you're just building some consumer app I'd say go for XML, as it saves you, and the consumer, all that trouble of installing a database. On modern machines size really doesn't matter.
Visit my blog at Sander's bits - Writing the code you need. Or read my articles at my CodeProject profile.
Simplicity is prerequisite for reliability. — Edsger W. Dijkstra
Regards, Sander
-
There are no database rockstars, but those who master databases are a SELECT group.
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
-
Databases do lots more than just store data. It's a centralized point for transactional data storage, modification and retrieval. Try using some XML files in an enterprise environment. That's simply not going to work. If you're just building some consumer app I'd say go for XML, as it saves you, and the consumer, all that trouble of installing a database. On modern machines size really doesn't matter.
Visit my blog at Sander's bits - Writing the code you need. Or read my articles at my CodeProject profile.
Simplicity is prerequisite for reliability. — Edsger W. Dijkstra
Regards, Sander
Even for a consumer app, it's not so cut and dried. I'd put the XML / DB cutoff as a table complexity and INSERT / UPDATE / DELETE criteria myself. XML is text based, so frequent changes are going to mean a lot of file copying. Even for mobile there is SQLCE or SqLite instead, both of which have a relatively small footprint.
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
-
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
It looks as I am BETWEEN monsters, WHERE was my head when I decided to JOIN? I'm about to DROP the USE of this GROUP BY heaven's sake! BTW I dislike SQL and DBs
Geek code v 3.12 GCS d--- s-/++ a- C++++ U+++ P- L- E-- W++ N++ o+ K- w+++ O? M-- V? PS+ PE- Y+ PGP t++ 5? X R++ tv-- b+ DI+++ D++ G e++>+++ h--- r++>+++ y+++* Weapons extension: ma- k++ F+2 X I use 1TBS
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
For me the question is not quite right. I don't think row count, or class count should determine whether a database is used or not. I think it's more a case of what you are going to do with the data. If you think you may need to 'look' at the data occasionally, manipulate it or run some form of reports(statistics, counts, aggregation...) on the data than a database is the right choice even if you have as few as 100 rows. If you are just reading in data that has already been 'cleaned' and all you are doing is transforming the data then perhaps a database is not necessary. As others have mentioned there is no correct answer - whichever path you take there will be advantages and disadvantages. My rule of thumb is not so much the volume of data, but what is going to happen to the data. Regarding footprint - there are some databases which leave a reasonably small(suitably vague statement of the day...) footprint, particularly those where the database is structured as one file.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
10,000 is very low number to compare anything for performance. When you compare XML / Flat file with database the only advantage you get is write performance and cheaper cost. Which is never a deciding factor for any application, whether its enterprise or consumer. The things you can get with database and not with xml are: - Concurrency (You can control if you have insert/update at the same time on your data) - Transaction Reliability (You can go back to an older data/snapshot to correct a mistake) - Backup (You can configure very efficient backup policy i.e. log shipping / replication which is not possible with XML) - Partitioning (You can store data to different database/location based on region and then retrieve it easily) - Archiving (Transfer data to archive in realtime and then access it in realtime with linked database) - Security (fine grain control over who can access what) - And the most important, Well Structured and Connected Data (makes complex query for different reports very easy) So I would suggest you to use XML for intermediate stuff, or to store settings. but other than that don't waste your time and efforts. Just go with the RDBMS. And for Mobile device to store offline stuff you can use XML / JSON Documents (JSON is better than XML). But at the end you should have a RDBMS server and Sync your offline data to it.
Remind Me This - Manage, Collaborate and Execute your Project in the Cloud
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
I'd go for XML with anything that is just a few lines, like configuration. If it contains a lot of relations, I'd switch to SQLite. If multiple users are to access the datasource, it would also become a real database. I don't want to have to implement things like table-locking.
Bastard Programmer from Hell :suss: If you can't read my code, try converting it here[^][](X-Clacks-Overhead: GNU Terry Pratchett)
-
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
Try using the KEY (the whole key, and nothing but the key)
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
If you need any kind of concurrency it can quickly become messy with (xml)files.
Wrong is evil and must be defeated. - Jeff Ello
-
Try using the KEY (the whole key, and nothing but the key)
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
It depends on the structure of the data. For an example of why this is important, if look at the structure of CodeProject, as you should have before you posted, you will see that you have inserted a Message data item to the Lounge table, which is not where it belongs in the structure. The effect of this is that stored procedures set up to resolve the type of problem in your Message object will not be able to find it with their pre-configured queries, so your problem will not be found, and will not be resolved. The world will not end and civilisation will not collapse because of this (probably, but don't quote me if it happens), but it does demonstrate that if the data you have to handle has a complex structure, or many different branches, then storing it in large XML files, e.g. pages of the Lounge, is a pretty damned stupid thing to do. ... Almost as stupid as posting programming questions in the CP Lounge.
I wanna be a eunuchs developer! Pass me a bread knife!
-
Try using the KEY (the whole key, and nothing but the key)
He could try, but he is not the right CANDIDATE to have the KEY.
You have just been Sharapova'd.
-
He could try, but he is not the right CANDIDATE to have the KEY.
You have just been Sharapova'd.
I think the nearest you can get to a rockstar DBA is Bobby McFerrin[^]
-
There are no database rockstars, but those who master databases are a SELECT group.
Yup, and we hang out in HEAPs and CLUSTERS.
If it's not broken, fix it until it is
-
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
If you're not clever enough to join the
RANKs[^], you could try theDENSE_RANKs[^]. ;P
"These people looked deep within my soul and assigned me a number based on the order in which I joined." - Homer
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
wreckless wrote:
Please enlighten me on the subject.
Data Integrity
Jeremy Falcon
