Work challenge of the day
-
At work, we have time series data collected from multiple remote sites. On occasion, a communication break with the remote site occurs and collected data is buffered; when the communication break is resolved the buffered data is passed to the receiving system. All of the data is time stamped from when it originated, not from when it is received at corporate. There are two types of calculations: event based - execute the calculation when an input event is received and time stamp the resultant value with the originating event time stamp periodic - execute the calculation on a timed basis and use whatever values are available; the resultant value is time stamped with the execution time So.. now the issue.. if the data is buffered, the periodic calculations are useless, but, periodic calculations are a business requirement. I need to devise a system to: detect a communication break detect when the communication break is resolved and the buffered data has been received delete the periodic values for the affected site for the affected time frame re-execute the periodic calculations for the affected site for the affected time frame keeping in mind calculation dependencies; an output from periodic calculation 'A' may be used in periodic calculation 'B' make the system that gathers the periodic data aware of the recalculated data so they can reload it I have a conceptual plan written... starting to work on the pieces.
-
At work, we have time series data collected from multiple remote sites. On occasion, a communication break with the remote site occurs and collected data is buffered; when the communication break is resolved the buffered data is passed to the receiving system. All of the data is time stamped from when it originated, not from when it is received at corporate. There are two types of calculations: event based - execute the calculation when an input event is received and time stamp the resultant value with the originating event time stamp periodic - execute the calculation on a timed basis and use whatever values are available; the resultant value is time stamped with the execution time So.. now the issue.. if the data is buffered, the periodic calculations are useless, but, periodic calculations are a business requirement. I need to devise a system to: detect a communication break detect when the communication break is resolved and the buffered data has been received delete the periodic values for the affected site for the affected time frame re-execute the periodic calculations for the affected site for the affected time frame keeping in mind calculation dependencies; an output from periodic calculation 'A' may be used in periodic calculation 'B' make the system that gathers the periodic data aware of the recalculated data so they can reload it I have a conceptual plan written... starting to work on the pieces.
You also need a system to devise what is the appropriate forum to ask these questions on :)
-
At work, we have time series data collected from multiple remote sites. On occasion, a communication break with the remote site occurs and collected data is buffered; when the communication break is resolved the buffered data is passed to the receiving system. All of the data is time stamped from when it originated, not from when it is received at corporate. There are two types of calculations: event based - execute the calculation when an input event is received and time stamp the resultant value with the originating event time stamp periodic - execute the calculation on a timed basis and use whatever values are available; the resultant value is time stamped with the execution time So.. now the issue.. if the data is buffered, the periodic calculations are useless, but, periodic calculations are a business requirement. I need to devise a system to: detect a communication break detect when the communication break is resolved and the buffered data has been received delete the periodic values for the affected site for the affected time frame re-execute the periodic calculations for the affected site for the affected time frame keeping in mind calculation dependencies; an output from periodic calculation 'A' may be used in periodic calculation 'B' make the system that gathers the periodic data aware of the recalculated data so they can reload it I have a conceptual plan written... starting to work on the pieces.
Never begin a new project on a Friday.
-
At work, we have time series data collected from multiple remote sites. On occasion, a communication break with the remote site occurs and collected data is buffered; when the communication break is resolved the buffered data is passed to the receiving system. All of the data is time stamped from when it originated, not from when it is received at corporate. There are two types of calculations: event based - execute the calculation when an input event is received and time stamp the resultant value with the originating event time stamp periodic - execute the calculation on a timed basis and use whatever values are available; the resultant value is time stamped with the execution time So.. now the issue.. if the data is buffered, the periodic calculations are useless, but, periodic calculations are a business requirement. I need to devise a system to: detect a communication break detect when the communication break is resolved and the buffered data has been received delete the periodic values for the affected site for the affected time frame re-execute the periodic calculations for the affected site for the affected time frame keeping in mind calculation dependencies; an output from periodic calculation 'A' may be used in periodic calculation 'B' make the system that gathers the periodic data aware of the recalculated data so they can reload it I have a conceptual plan written... starting to work on the pieces.
Damn you! I am sat here at work with nothing to do but wait for any problems to rise and I've had one support call today and new development projects start next year :(
Every day, thousands of innocent plants are killed by vegetarians. Help end the violence EAT BACON
-
At work, we have time series data collected from multiple remote sites. On occasion, a communication break with the remote site occurs and collected data is buffered; when the communication break is resolved the buffered data is passed to the receiving system. All of the data is time stamped from when it originated, not from when it is received at corporate. There are two types of calculations: event based - execute the calculation when an input event is received and time stamp the resultant value with the originating event time stamp periodic - execute the calculation on a timed basis and use whatever values are available; the resultant value is time stamped with the execution time So.. now the issue.. if the data is buffered, the periodic calculations are useless, but, periodic calculations are a business requirement. I need to devise a system to: detect a communication break detect when the communication break is resolved and the buffered data has been received delete the periodic values for the affected site for the affected time frame re-execute the periodic calculations for the affected site for the affected time frame keeping in mind calculation dependencies; an output from periodic calculation 'A' may be used in periodic calculation 'B' make the system that gathers the periodic data aware of the recalculated data so they can reload it I have a conceptual plan written... starting to work on the pieces.
delete from ALL_PERIODIC_DATA
sp_reconstructAllPeriodicData();That's the way we'd do it here. :sigh:
Latest Article - Class-less Coding - Minimalist C# and Why F# and Function Programming Has Some Advantages Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
-
You also need a system to devise what is the appropriate forum to ask these questions on :)
Not asking a question; just stating what I'm working on.
-
Never begin a new project on a Friday.
I've been mulling it over in my mind since about August; just now have time to actually start to dump the thoughts.
-
delete from ALL_PERIODIC_DATA
sp_reconstructAllPeriodicData();That's the way we'd do it here. :sigh:
Latest Article - Class-less Coding - Minimalist C# and Why F# and Function Programming Has Some Advantages Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
SQL Server is not an option... the calculation engine is part of a vendor supplied package. But... it is a great challenge!
-
At work, we have time series data collected from multiple remote sites. On occasion, a communication break with the remote site occurs and collected data is buffered; when the communication break is resolved the buffered data is passed to the receiving system. All of the data is time stamped from when it originated, not from when it is received at corporate. There are two types of calculations: event based - execute the calculation when an input event is received and time stamp the resultant value with the originating event time stamp periodic - execute the calculation on a timed basis and use whatever values are available; the resultant value is time stamped with the execution time So.. now the issue.. if the data is buffered, the periodic calculations are useless, but, periodic calculations are a business requirement. I need to devise a system to: detect a communication break detect when the communication break is resolved and the buffered data has been received delete the periodic values for the affected site for the affected time frame re-execute the periodic calculations for the affected site for the affected time frame keeping in mind calculation dependencies; an output from periodic calculation 'A' may be used in periodic calculation 'B' make the system that gathers the periodic data aware of the recalculated data so they can reload it I have a conceptual plan written... starting to work on the pieces.
-
video recordings? ... you need some protocol to detect when something when offline and log it?
Caveat Emptor. "Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long
We have essentially two different 'types' of data feeds; for one, we can detect that we are not receiving data, and upon resumption, how much data is buffered. However, the other type is more problematic: it is largely a 'copy from here to here' system that doesn't report how much data is waiting to be sent... so my challenge will be to find out how to detect the break and resumption in the other system. Again, just stating what my challenge is. Many of these sites have low band width and video would overwhelm the system.
-
We have essentially two different 'types' of data feeds; for one, we can detect that we are not receiving data, and upon resumption, how much data is buffered. However, the other type is more problematic: it is largely a 'copy from here to here' system that doesn't report how much data is waiting to be sent... so my challenge will be to find out how to detect the break and resumption in the other system. Again, just stating what my challenge is. Many of these sites have low band width and video would overwhelm the system.
-
Not asking a question; just stating what I'm working on.
-
No procrastinating, just finding time to start on it.
