One for the math/stats buffs.
-
What you might want to do is outlyer detection, and just encode those results with some kind of symbol on the chart like a red line or something like that. Or, as others have said, just drop them and leave them as gaps. Also, you can compute the median and standard deviation for your series, and then just plot everything within a few standard deviations of the median, which should be pretty close to the same thing.
Curvature of the Mind now with 3D
How do I calculate the median. That would be a straight line - I want a 'median' between lows and highs that moves up and down with the lows and highs. I suppose I could do it point for point and calculate a median curve.
-
How do I calculate the median. That would be a straight line - I want a 'median' between lows and highs that moves up and down with the lows and highs. I suppose I could do it point for point and calculate a median curve.
Ok, that gets into how you display the data. For a single static graph, I'd do it for the entire graph. That would give the bounds for the entire image. If I was calculating a rolling average, I'd plot that as a curve overlay on top of the existing data. Check out various stock charts for that kind of visualization. To calculate a rolling average, you just assign a window to each point, for example the last 30 data points and calculate the statistics on that window. The trick is really to use a window that makes sense for the underlying data.
Curvature of the Mind now with 3D
-
How do I calculate the median. That would be a straight line - I want a 'median' between lows and highs that moves up and down with the lows and highs. I suppose I could do it point for point and calculate a median curve.
Also, if the y scale varies across the width of the graph according to the data, that is going to be very difficult to interpret, as you will be "straightening" out the data and none of the variation will be absolute. Sometimes it's just easier to us a log scale for wildly varying data.
Curvature of the Mind now with 3D
-
Also, if the y scale varies across the width of the graph according to the data, that is going to be very difficult to interpret, as you will be "straightening" out the data and none of the variation will be absolute. Sometimes it's just easier to us a log scale for wildly varying data.
Curvature of the Mind now with 3D
It's not at all widely varying. It's mains voltage over the day, and most samples vary by7 a few volts on 240V. It's juts a very few funny points that I think I will just ignore. You can't really depend ona graph as an accurate source of data, and I have a report of all the exact values.
-
Ok, that gets into how you display the data. For a single static graph, I'd do it for the entire graph. That would give the bounds for the entire image. If I was calculating a rolling average, I'd plot that as a curve overlay on top of the existing data. Check out various stock charts for that kind of visualization. To calculate a rolling average, you just assign a window to each point, for example the last 30 data points and calculate the statistics on that window. The trick is really to use a window that makes sense for the underlying data.
Curvature of the Mind now with 3D
By a window, you mean calculate the average low for 30 points, the same for high, and the mean of those is one point in your running central mean curve?
-
I have a range of values (voltage) over time (thousands of minutes, one value per minute). I am trying to chart these. Determining the length of my Y axis is quite a problem for me. If I take a minimum and maximum, and use that as the axis height, one or two zero values result in all the others being scrunched up at the top of the chart. If I remove zeroes, it looks much better, and for a chart, they aren't very important, I'll give all real values in a tabular report. What I would like to do is determine the average height of the band of data points, sort of the space between the moving average of the low points and that of the heigh points. I figure to do that, I would need a median series, so I could determine a smoothed series of points above and below median, and make my Y axis 's' higher and 's' lower than those. How do people normally do this?
Personally I would start in Excel - just to be able to play easily with various stats functions and see the result easily. To determine your Y bounds: Since we are looking at mains voltage, the maximum is of interest, so your Y max should be based on the maximum of your data. Take the average M. Calculate the standard deviation s. In Excel take M - N_s_ for the lower bound - play with N to find a reasonable value. I suspect that 3 will probably work. Now extract all those points below the lower bound and put them aside for later. Now take the minimum of the remaining points round it to the nearest 5, 10 etc. based on the final range that you use. Round the maximum in the same way. (Note: depending on the measuring equipment you might get spurious high values too, you can do the same with the maximum bound if you want, but might need a different value for N) This should give you the plot that you want. Now, those pesky low values.... These should figure as points along the bottom of the graph, downward pointing arrows would be nice, since we need to be scientifically correct and show that the data has been "massaged". Set their value to Ymin plus 7% of the Y range for example. Hope this helps
-
I have a range of values (voltage) over time (thousands of minutes, one value per minute). I am trying to chart these. Determining the length of my Y axis is quite a problem for me. If I take a minimum and maximum, and use that as the axis height, one or two zero values result in all the others being scrunched up at the top of the chart. If I remove zeroes, it looks much better, and for a chart, they aren't very important, I'll give all real values in a tabular report. What I would like to do is determine the average height of the band of data points, sort of the space between the moving average of the low points and that of the heigh points. I figure to do that, I would need a median series, so I could determine a smoothed series of points above and below median, and make my Y axis 's' higher and 's' lower than those. How do people normally do this?
Since the zero readings are anomalies, then I would exclude them from the chart data (thereby cleaning up the chart display), and keep a separate record of the zero readings viewable in a table with time of reading. That way you get a clean chart & a separate table of anomalous values if you need to look at them. Or you could chart the zero readings in another way - number of zero readings per hour against time perhaps?
-
By a window, you mean calculate the average low for 30 points, the same for high, and the mean of those is one point in your running central mean curve?
for the 30 point window it would be just calculate the stats for n-30 to n, or n-15 to n+15, or whatever window makes sense and graph those values for n. It's just a sliding window to do your calculations.
Curvature of the Mind now with 3D
-
It gets quite sweet when I exclude the zeroes. The chart then spreads across the Y axis nicely.
Brady Kelly wrote:
It gets quite sweet when I exclude the zeroes. The chart then spreads across the Y axis nicely
Sounds like you want an arbitrary axis scale. Assuming you're using Excel, here's a link to a page on how to do that. I've not had occasion to use it, and it's trying to make a Microsoft tool do something that it's not designed to do, which as we all know is fraught with pain.
We can program with only 1's, but if all you've got are zeros, you've got nothing.
-
I have a range of values (voltage) over time (thousands of minutes, one value per minute). I am trying to chart these. Determining the length of my Y axis is quite a problem for me. If I take a minimum and maximum, and use that as the axis height, one or two zero values result in all the others being scrunched up at the top of the chart. If I remove zeroes, it looks much better, and for a chart, they aren't very important, I'll give all real values in a tabular report. What I would like to do is determine the average height of the band of data points, sort of the space between the moving average of the low points and that of the heigh points. I figure to do that, I would need a median series, so I could determine a smoothed series of points above and below median, and make my Y axis 's' higher and 's' lower than those. How do people normally do this?
If you want to show all the data you could try splitting the chart. Split the data in two and plot the graphs stacked. An ASCII representation of what you would aim for is:
250 V |-------------------------
240 V |-------------------------
230 V |-------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~
1 V |-------------------------
0 V |------------------------- -
I have a range of values (voltage) over time (thousands of minutes, one value per minute). I am trying to chart these. Determining the length of my Y axis is quite a problem for me. If I take a minimum and maximum, and use that as the axis height, one or two zero values result in all the others being scrunched up at the top of the chart. If I remove zeroes, it looks much better, and for a chart, they aren't very important, I'll give all real values in a tabular report. What I would like to do is determine the average height of the band of data points, sort of the space between the moving average of the low points and that of the heigh points. I figure to do that, I would need a median series, so I could determine a smoothed series of points above and below median, and make my Y axis 's' higher and 's' lower than those. How do people normally do this?
-
I have a range of values (voltage) over time (thousands of minutes, one value per minute). I am trying to chart these. Determining the length of my Y axis is quite a problem for me. If I take a minimum and maximum, and use that as the axis height, one or two zero values result in all the others being scrunched up at the top of the chart. If I remove zeroes, it looks much better, and for a chart, they aren't very important, I'll give all real values in a tabular report. What I would like to do is determine the average height of the band of data points, sort of the space between the moving average of the low points and that of the heigh points. I figure to do that, I would need a median series, so I could determine a smoothed series of points above and below median, and make my Y axis 's' higher and 's' lower than those. How do people normally do this?
Removing zeroes is not good practice unless you have a legitimate reason to discard them (e.g. the measuring device failed). Can't you use a log scale or simply calculate axis span to ignore the lowest 10% of values or those values more than 2 standard deviations below the mean or some other systematic criterion? As for the second question, can't you use some sort of best-fit curve along with an indication of range, such as standard deviation to determine lower and upper bounds, or better still the standard error of the mean?