Simple question for the Database Rock Stars
-
There are no database rockstars, but those who master databases are a SELECT group.
Yeah, it's pretty hard to JOIN their RANKS! :D
Visit my blog at Sander's bits - Writing the code you need. Or read my articles at my CodeProject profile.
Simplicity is prerequisite for reliability. — Edsger W. Dijkstra
Regards, Sander
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
Databases do lots more than just store data. It's a centralized point for transactional data storage, modification and retrieval. Try using some XML files in an enterprise environment. That's simply not going to work. If you're just building some consumer app I'd say go for XML, as it saves you, and the consumer, all that trouble of installing a database. On modern machines size really doesn't matter.
Visit my blog at Sander's bits - Writing the code you need. Or read my articles at my CodeProject profile.
Simplicity is prerequisite for reliability. — Edsger W. Dijkstra
Regards, Sander
-
There are no database rockstars, but those who master databases are a SELECT group.
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt -
Databases do lots more than just store data. It's a centralized point for transactional data storage, modification and retrieval. Try using some XML files in an enterprise environment. That's simply not going to work. If you're just building some consumer app I'd say go for XML, as it saves you, and the consumer, all that trouble of installing a database. On modern machines size really doesn't matter.
Visit my blog at Sander's bits - Writing the code you need. Or read my articles at my CodeProject profile.
Simplicity is prerequisite for reliability. — Edsger W. Dijkstra
Regards, Sander
Even for a consumer app, it's not so cut and dried. I'd put the XML / DB cutoff as a table complexity and INSERT / UPDATE / DELETE criteria myself. XML is text based, so frequent changes are going to mean a lot of file copying. Even for mobile there is SQLCE or SqLite instead, both of which have a relatively small footprint.
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
-
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
It looks as I am BETWEEN monsters, WHERE was my head when I decided to JOIN? I'm about to DROP the USE of this GROUP BY heaven's sake! BTW I dislike SQL and DBs
Geek code v 3.12 GCS d--- s-/++ a- C++++ U+++ P- L- E-- W++ N++ o+ K- w+++ O? M-- V? PS+ PE- Y+ PGP t++ 5? X R++ tv-- b+ DI+++ D++ G e++>+++ h--- r++>+++ y+++* Weapons extension: ma- k++ F+2 X I use 1TBS
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
For me the question is not quite right. I don't think row count, or class count should determine whether a database is used or not. I think it's more a case of what you are going to do with the data. If you think you may need to 'look' at the data occasionally, manipulate it or run some form of reports(statistics, counts, aggregation...) on the data than a database is the right choice even if you have as few as 100 rows. If you are just reading in data that has already been 'cleaned' and all you are doing is transforming the data then perhaps a database is not necessary. As others have mentioned there is no correct answer - whichever path you take there will be advantages and disadvantages. My rule of thumb is not so much the volume of data, but what is going to happen to the data. Regarding footprint - there are some databases which leave a reasonably small(suitably vague statement of the day...) footprint, particularly those where the database is structured as one file.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
10,000 is very low number to compare anything for performance. When you compare XML / Flat file with database the only advantage you get is write performance and cheaper cost. Which is never a deciding factor for any application, whether its enterprise or consumer. The things you can get with database and not with xml are: - Concurrency (You can control if you have insert/update at the same time on your data) - Transaction Reliability (You can go back to an older data/snapshot to correct a mistake) - Backup (You can configure very efficient backup policy i.e. log shipping / replication which is not possible with XML) - Partitioning (You can store data to different database/location based on region and then retrieve it easily) - Archiving (Transfer data to archive in realtime and then access it in realtime with linked database) - Security (fine grain control over who can access what) - And the most important, Well Structured and Connected Data (makes complex query for different reports very easy) So I would suggest you to use XML for intermediate stuff, or to store settings. but other than that don't waste your time and efforts. Just go with the RDBMS. And for Mobile device to store offline stuff you can use XML / JSON Documents (JSON is better than XML). But at the end you should have a RDBMS server and Sync your offline data to it.
Remind Me This - Manage, Collaborate and Execute your Project in the Cloud
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
I'd go for XML with anything that is just a few lines, like configuration. If it contains a lot of relations, I'd switch to SQLite. If multiple users are to access the datasource, it would also become a real database. I don't want to have to implement things like table-locking.
Bastard Programmer from Hell :suss: If you can't read my code, try converting it here[^][](X-Clacks-Overhead: GNU Terry Pratchett)
-
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
Try using the KEY (the whole key, and nothing but the key)
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
If you need any kind of concurrency it can quickly become messy with (xml)files.
Wrong is evil and must be defeated. - Jeff Ello
-
Try using the KEY (the whole key, and nothing but the key)
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
It depends on the structure of the data. For an example of why this is important, if look at the structure of CodeProject, as you should have before you posted, you will see that you have inserted a Message data item to the Lounge table, which is not where it belongs in the structure. The effect of this is that stored procedures set up to resolve the type of problem in your Message object will not be able to find it with their pre-configured queries, so your problem will not be found, and will not be resolved. The world will not end and civilisation will not collapse because of this (probably, but don't quote me if it happens), but it does demonstrate that if the data you have to handle has a complex structure, or many different branches, then storing it in large XML files, e.g. pages of the Lounge, is a pretty damned stupid thing to do. ... Almost as stupid as posting programming questions in the CP Lounge.
I wanna be a eunuchs developer! Pass me a bread knife!
-
Try using the KEY (the whole key, and nothing but the key)
He could try, but he is not the right CANDIDATE to have the KEY.
You have just been Sharapova'd.
-
He could try, but he is not the right CANDIDATE to have the KEY.
You have just been Sharapova'd.
I think the nearest you can get to a rockstar DBA is Bobby McFerrin[^]
-
There are no database rockstars, but those who master databases are a SELECT group.
Yup, and we hang out in HEAPs and CLUSTERS.
If it's not broken, fix it until it is
-
Indeed, a GROUP HAVING few members. I'd INSERT myself in the ranks, but I am not worthy to JOIN...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
If you're not clever enough to join the
RANKs[^], you could try theDENSE_RANKs[^]. ;P
"These people looked deep within my soul and assigned me a number based on the order in which I joined." - Homer
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
wreckless wrote:
Please enlighten me on the subject.
Data Integrity
Jeremy Falcon
-
There are no database rockstars, but those who master databases are a SELECT group.
here we go.... :sigh:
Charlie Gilley Stuck in a dysfunctional matrix from which I must escape... "Where liberty dwells, there is my country." B. Franklin, 1783 “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
-
Just kidding about the rock stars but thought it might get some attention. But I do have a question. I have read all about the databases and really don't care which one is better or more efficient and so on, what I want to know and have always wanted to ask: If you have a simple program that needs to store data, what is the data size cut off that it is more efficient to use a database or just serialize to files (xml or other)? I have tested xml files holding as many as 10,000 elements, and have not noticed much of a delay putting same elements in database tables. I have tried several different approaches and tried to complicate the tables and queries and didn't see much difference in performance. Has anybody ever done any load testing or any research on what would be the cutoff point on data size, or evaluated the overhead over XML or a Database? I also know that there is a lot of support for working with databases and cloud storage which simplifies the DAL & LOB data access and storage, but it also leaves a large footprint on your device for something where the data load might not be so large as to warrant such a large footprint, such as mobile devices. Or does it? I don't know that's why I am asking. Please enlighten me on the subject.
The difference between a question and a stupid question depends on who you ask to answer it! wreckless
wreckless wrote:
Just kidding about the rock stars
Good, 'cause anyone who thinks he's a rock star is actually just another hipster douchebag. It's not about size. Others have pointed out some of the many concerns involved. In short, XML is fine for configuration and reports, but not suitable for random access and multi-user access.
-
wreckless wrote:
Just kidding about the rock stars
Good, 'cause anyone who thinks he's a rock star is actually just another hipster douchebag. It's not about size. Others have pointed out some of the many concerns involved. In short, XML is fine for configuration and reports, but not suitable for random access and multi-user access.
I knew I should have left the rock stars part out of it. I also learned it's not a good idea to ask a question when I've been drinking. It was just spontaneous combustion I guess. But what the heck, sh#t happens. (Like posting in the lounge, sorry bout that) Lots of good answers here anyway, It looks to me like if it is simple keep it simple. If it is complicated, keep it simple by using a database. That about sums it up. But then again, how do you define simple? What simple for some may be Rock Science to others.
I guess it all depends on what is is? wreckless
