Persistent storage of measuring results
-
Hello, I want to store the recent measuring results as well as control samples of older measurements persitently on disk so that the last state can be restored even in the case of a power loss. To make loading the data on startup not unnecessary slow, I want to remove the old, unneeded data from the list. This means that the measuring results will be continuously appended to the list, but the time when it will be removed is not known in advance. Would you recommend using a database (ideally some compact system like SQLite so that the user does not have to install a databse server) for this, or are there alternatives? My idea would be to store the measuring results in one table, using the measuring time as key. In another table, the keys for the last n measuring results would be stored and in a third table the keys of the control samples. If the entries of the first table are not referenced any more from the two other tables, the measuring results should be automatically removed. Is there some concept to acchieve this? Alex
-
Hello, I want to store the recent measuring results as well as control samples of older measurements persitently on disk so that the last state can be restored even in the case of a power loss. To make loading the data on startup not unnecessary slow, I want to remove the old, unneeded data from the list. This means that the measuring results will be continuously appended to the list, but the time when it will be removed is not known in advance. Would you recommend using a database (ideally some compact system like SQLite so that the user does not have to install a databse server) for this, or are there alternatives? My idea would be to store the measuring results in one table, using the measuring time as key. In another table, the keys for the last n measuring results would be stored and in a third table the keys of the control samples. If the entries of the first table are not referenced any more from the two other tables, the measuring results should be automatically removed. Is there some concept to acchieve this? Alex
This decision should be driven by volume and usage. If you are only processing a few hundred records and do not need to massage the data then an XML or csv file store would be acceptable. Otherwise go for a database. Why make it complex with multiple tables if you are using text/xml, simply load the entire file into memory and manipulate the collection. Write the collection to the file at regular intervals. A lot will depend on the criticallity of the data and the frequency of the reads!
Never underestimate the power of human stupidity RAH
-
This decision should be driven by volume and usage. If you are only processing a few hundred records and do not need to massage the data then an XML or csv file store would be acceptable. Otherwise go for a database. Why make it complex with multiple tables if you are using text/xml, simply load the entire file into memory and manipulate the collection. Write the collection to the file at regular intervals. A lot will depend on the criticallity of the data and the frequency of the reads!
Never underestimate the power of human stupidity RAH
Thank you for your answer. In a previous version, we wrote the complete XML file - however, due to the relative large amount of data (and the limited resources, as it is an embedded system) - this will not be possible in the final release. So I think the database is the most promising. Which leaves me with the other questions, how to implement this as a weak reference pattern. Alex
-
Thank you for your answer. In a previous version, we wrote the complete XML file - however, due to the relative large amount of data (and the limited resources, as it is an embedded system) - this will not be possible in the final release. So I think the database is the most promising. Which leaves me with the other questions, how to implement this as a weak reference pattern. Alex
LionAM wrote:
how to implement this as a weak reference pattern.
Databases generally do not directly support the equivalent of weak reference pattern. One solution is to append the data as needed in a database table and then run a program in background periodically and check for "old" data and purge them from the database. What "old" means and the frequency of purging depends on the volume of your data.
-
Hello, I want to store the recent measuring results as well as control samples of older measurements persitently on disk so that the last state can be restored even in the case of a power loss. To make loading the data on startup not unnecessary slow, I want to remove the old, unneeded data from the list. This means that the measuring results will be continuously appended to the list, but the time when it will be removed is not known in advance. Would you recommend using a database (ideally some compact system like SQLite so that the user does not have to install a databse server) for this, or are there alternatives? My idea would be to store the measuring results in one table, using the measuring time as key. In another table, the keys for the last n measuring results would be stored and in a third table the keys of the control samples. If the entries of the first table are not referenced any more from the two other tables, the measuring results should be automatically removed. Is there some concept to acchieve this? Alex
As a database kind of guy, I strongly recommend a database.
LionAM wrote:
To make loading the data on startup not unnecessary slow
I asssume you do this to show a graph or similar of the recent trend, otherwise I don't see why you would load old data. A database makes this easy by allowing you to query only the recent data. You could sort the data descending by date and take only the top n records or you could query the records where the timestamp is greater than the current time minus soome number of minutes or hours. Or you could purge data out of the table after some period (perhaps after summarizing it into another for historical reasons). Also, as I don't know what sorts of "measuring results" you are dealing with I'll pass along a concept that you may find useful. It's sort of a data compression technique whereby if the measurements don't change much you don't store every individual measurement. For instance, the temperature of a commercial oven for baking bread or cookies needs to be monitored, it can probably vary a little (let's say a tenth of a degree) from the target temperature without causing an alarm, but if it changes a whole degree (for instance) it should be noted. Furthermore, if the temperature stays close to the target temperature for hours on end (as it should) we do want to log some readings (perhaps at least every fifteen minutes). For these reasons, you may want to develop a system that will only store a reading if the difference from the previous stored reading is outside some specified range or if some specified time span has elapsed.
-
As a database kind of guy, I strongly recommend a database.
LionAM wrote:
To make loading the data on startup not unnecessary slow
I asssume you do this to show a graph or similar of the recent trend, otherwise I don't see why you would load old data. A database makes this easy by allowing you to query only the recent data. You could sort the data descending by date and take only the top n records or you could query the records where the timestamp is greater than the current time minus soome number of minutes or hours. Or you could purge data out of the table after some period (perhaps after summarizing it into another for historical reasons). Also, as I don't know what sorts of "measuring results" you are dealing with I'll pass along a concept that you may find useful. It's sort of a data compression technique whereby if the measurements don't change much you don't store every individual measurement. For instance, the temperature of a commercial oven for baking bread or cookies needs to be monitored, it can probably vary a little (let's say a tenth of a degree) from the target temperature without causing an alarm, but if it changes a whole degree (for instance) it should be noted. Furthermore, if the temperature stays close to the target temperature for hours on end (as it should) we do want to log some readings (perhaps at least every fifteen minutes). For these reasons, you may want to develop a system that will only store a reading if the difference from the previous stored reading is outside some specified range or if some specified time span has elapsed.
In principle, this is exactly what I want - store a live view of a certain time span as well as the status of several control charts. As the statistical spread of the measurement results is of great interest, it makes no sense to store only large fluctuations (then it would perhaps be too late to countersteer ...). As the data is only used to restore the last program state and therefore stored locally (there is a separate export feature), I would prefer some compact database (without a database server). Which would you recommend? SQLite? Alex
-
In principle, this is exactly what I want - store a live view of a certain time span as well as the status of several control charts. As the statistical spread of the measurement results is of great interest, it makes no sense to store only large fluctuations (then it would perhaps be too late to countersteer ...). As the data is only used to restore the last program state and therefore stored locally (there is a separate export feature), I would prefer some compact database (without a database server). Which would you recommend? SQLite? Alex
LionAM wrote:
it makes no sense to store only large fluctuations
You can set the thresholds as appropriate.
LionAM wrote:
some compact database
I've dabbled in Sql Server Compact Edition recently, but I can't really give an informed recommendation. You could also consider serializing the current state to a file periodically. Maybe my LimitedQueue[^] could be of use to you.
-
In principle, this is exactly what I want - store a live view of a certain time span as well as the status of several control charts. As the statistical spread of the measurement results is of great interest, it makes no sense to store only large fluctuations (then it would perhaps be too late to countersteer ...). As the data is only used to restore the last program state and therefore stored locally (there is a separate export feature), I would prefer some compact database (without a database server). Which would you recommend? SQLite? Alex
LionAM wrote:
SQLite?
I'd recommend programming against the
IDbInterfaces, and actually time/test the various databases that you would want to consider.- Sqlite is good in terms of performance, once you take in account that everything is in a transaction.
- SqlCe is more integrated into .NET, and would be the preferred option if you plan to synchronize to Sql Server.
- Microsoft Access would be a sweet idea if your client has Office, and if the primary goal of the data is to show reports.
- Excel would work too, almost as fast as Access, and can be accessed over ADO.NET; Not an obvious choice as a datastore, but still very nice for charts.
Bastard Programmer from Hell :suss: if you can't read my code, try converting it here[^]
-
LionAM wrote:
SQLite?
I'd recommend programming against the
IDbInterfaces, and actually time/test the various databases that you would want to consider.- Sqlite is good in terms of performance, once you take in account that everything is in a transaction.
- SqlCe is more integrated into .NET, and would be the preferred option if you plan to synchronize to Sql Server.
- Microsoft Access would be a sweet idea if your client has Office, and if the primary goal of the data is to show reports.
- Excel would work too, almost as fast as Access, and can be accessed over ADO.NET; Not an obvious choice as a datastore, but still very nice for charts.
Bastard Programmer from Hell :suss: if you can't read my code, try converting it here[^]
Eddy Vluggen wrote:
xcel would work too, almost as fast as Access, and can be accessed over ADO.NET; Not an obvious choice as a datastore, but still very nice for charts.
And if the primary goal of the data is to be accessed by accountants. :)
Light moves faster than sound. That is why some people appear bright, until you hear them speak. List of common misconceptions
-
LionAM wrote:
SQLite?
I'd recommend programming against the
IDbInterfaces, and actually time/test the various databases that you would want to consider.- Sqlite is good in terms of performance, once you take in account that everything is in a transaction.
- SqlCe is more integrated into .NET, and would be the preferred option if you plan to synchronize to Sql Server.
- Microsoft Access would be a sweet idea if your client has Office, and if the primary goal of the data is to show reports.
- Excel would work too, almost as fast as Access, and can be accessed over ADO.NET; Not an obvious choice as a datastore, but still very nice for charts.
Bastard Programmer from Hell :suss: if you can't read my code, try converting it here[^]