MongoDB to Cache MariaDB?
-
Eddy Vluggen wrote:
Depending on your network, it may be very expensive to fetch a large blob (say 250 Mb on a 10 Mb network). In that case you mimick IIS - you fetch a picture once, and if it's fetched again you send a HTML 304.
Shouldn't be a big problem, since I plan on using the database locally.
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
If you plan on using it locally you can just as well use SQLite, concurrency won't be a problem then. And if you plan on moving it to a separate server later you will anyway have to take network problems into consideration already now.
Wrong is evil and must be defeated. - Jeff Ello[^]
-
If you plan on using it locally you can just as well use SQLite, concurrency won't be a problem then. And if you plan on moving it to a separate server later you will anyway have to take network problems into consideration already now.
Wrong is evil and must be defeated. - Jeff Ello[^]
Jörgen Andersson wrote:
concurrency won't be a problem then.
It will be, since a user may run more than one instance of the software, or I'm going to use multiple threads to access the database.
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
-
Jörgen Andersson wrote:
concurrency won't be a problem then.
It will be, since a user may run more than one instance of the software, or I'm going to use multiple threads to access the database.
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
How many threads are we talking about, or rather, how many writes are we talking about? The problem SQLite has with concurrency is that it locks the file rather than the row or block when you're doing DML on it. But every operation takes only milliseconds, so timeouts aren't likely. It's hard for me to believe that you could overload it with the use you specified earlier.
Wrong is evil and must be defeated. - Jeff Ello[^]
-
Jörgen Andersson wrote:
concurrency won't be a problem then.
It will be, since a user may run more than one instance of the software, or I'm going to use multiple threads to access the database.
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
Marco Bertschi wrote:
It will be, since a user may run more than one instance of the software
Doesn't "have" to be; there might as well be a Windows-service that works as a DAL - and have your app interface with that. Still, for multiple users one would recommend a *server*-database, complete with a server. It'd be a bit overkill to add that functionality yourself. And yes, that will introduce network-latency, but still does not mean that you need a second server-instance at the client. Store cache the static data locally; that'd be the pictures. Generate a hash to see what's modified. Put the relational info in the server, and access that without caching. Lots of webapps have a webservice-layer on top of that DB and perform decent - so I'd assume that any app querying directly would be faster. You could then omit the client-db completely and cache the images on the filesystem.
Bastard Programmer from Hell :suss: If you can't read my code, try converting it here[^]
-
How many threads are we talking about, or rather, how many writes are we talking about? The problem SQLite has with concurrency is that it locks the file rather than the row or block when you're doing DML on it. But every operation takes only milliseconds, so timeouts aren't likely. It's hard for me to believe that you could overload it with the use you specified earlier.
Wrong is evil and must be defeated. - Jeff Ello[^]
Jörgen Andersson wrote:
It's hard for me to believe that you could overload it with the use you specified earlier.
No, that isn't the problem I see - But later there might be a Web service in front of the DB, and many async write operations may deadlock SQLite - As you said, MariaDB is the way to go.
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
-
Jörgen Andersson wrote:
It's hard for me to believe that you could overload it with the use you specified earlier.
No, that isn't the problem I see - But later there might be a Web service in front of the DB, and many async write operations may deadlock SQLite - As you said, MariaDB is the way to go.
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
Ah, there you go, MariaDB it is then.
Wrong is evil and must be defeated. - Jeff Ello[^]
-
Dear all, I'm currently trying to make a DB system scratch which I can use as backbone for my Qt application. Since I will store big amounts of data [photo RAW data] in there, I figured that a classic relational DB might not be the best solution. What I'm planning is now a classic relational DB [MariaDB] as persistent storage, cached by a No-SQL Database [MongoDB]. In the end the whole system shall be built up like
Marco Bertschi wrote:
Since I will store big amounts of data
Where "big" means that the image size is 100 gigs or 100k? And there are 1 billion images or 1000? What is the projected growth rate?
Marco Bertschi wrote:
I I'm in doubt whether the MongoDB will really give me any performance boosts
What makes you think that you need a performance boost? How many requests does your business model realistically require per second? What is the distribution of the type of requests within an average hour?
-
Jörgen Andersson wrote:
It's hard for me to believe that you could overload it with the use you specified earlier.
No, that isn't the problem I see - But later there might be a Web service in front of the DB, and many async write operations may deadlock SQLite - As you said, MariaDB is the way to go.
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
Marco Bertschi wrote:
But later there might be a Web service in front of the DB,
Just noting that you seem to be suggesting that you can write both a client application and a web service application and somehow everything will be the same. Although that can be done the complications involved are significant. Those complications seldom add anything to the functionality of either. And doing it successfully without prior experience is going to be a challenge as well.
-
Marco Bertschi wrote:
Since I will store big amounts of data
Where "big" means that the image size is 100 gigs or 100k? And there are 1 billion images or 1000? What is the projected growth rate?
Marco Bertschi wrote:
I I'm in doubt whether the MongoDB will really give me any performance boosts
What makes you think that you need a performance boost? How many requests does your business model realistically require per second? What is the distribution of the type of requests within an average hour?
jschell wrote:
Where "big" means that the image size is 100 gigs or 100k? And there are 1 billion images or 1000? What is the projected growth rate?
"Big" in relation to a single entry in a table - A single RAW image can easily add up to more than 10 MB (which is big in comparison to other data, e.g. plain text)..
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.
-
jschell wrote:
Where "big" means that the image size is 100 gigs or 100k? And there are 1 billion images or 1000? What is the projected growth rate?
"Big" in relation to a single entry in a table - A single RAW image can easily add up to more than 10 MB (which is big in comparison to other data, e.g. plain text)..
I will never again mention that Dalek Dave was the poster of the One Millionth Lounge Post, nor that it was complete drivel.