The rad trap (warning code but not a question)
-
Preface ... I knew better but I made the mistake anyway and I just felt like sharing, you probably don't want to read : ) My current mission is to author applications that have never been written before, ever, rapidly, for the power industry on "Massive Amounts" of data for users that want it yesterday but will change their minds constantly. So pretend you want to know how long a users power was out (and I know ahead of time the rules will change on me)
public class Foo{
public DateTime PowerOutTime;
public DateTime PowerOnTime;
public TimeSpan FigureOutJustHowGDLongThePowerWasOut;
public string CustomerId;
}Populate the list with a few hundred million records, do some work. Output, slow, of course, data-loading takes a long time, up to 60 to 100 seconds for a day. Ignore the fact that I can't do this work in the DB (there are reasons) First naive mistake, Let's put it in a generic List
List items = ...
Sure the data calculation part is fast but for technical reasons this is very slow, (Allocation of contigous memory streams, copying, adding etc). Even better, when you are really sloppy and work with 7 days of customer data System.OutOfMemoryException : ). So what to do? Still blinded by the R.A.D. issue I continued ...
LinkedList items = ...
Golden, no longer an out of memory issue for n "Luke, pretend this is C++ and you are using a struct and a pointer offset" "Ben, I can't C# doesn't allow safe casting" "Use the force" So I map
-
Preface ... I knew better but I made the mistake anyway and I just felt like sharing, you probably don't want to read : ) My current mission is to author applications that have never been written before, ever, rapidly, for the power industry on "Massive Amounts" of data for users that want it yesterday but will change their minds constantly. So pretend you want to know how long a users power was out (and I know ahead of time the rules will change on me)
public class Foo{
public DateTime PowerOutTime;
public DateTime PowerOnTime;
public TimeSpan FigureOutJustHowGDLongThePowerWasOut;
public string CustomerId;
}Populate the list with a few hundred million records, do some work. Output, slow, of course, data-loading takes a long time, up to 60 to 100 seconds for a day. Ignore the fact that I can't do this work in the DB (there are reasons) First naive mistake, Let's put it in a generic List
List items = ...
Sure the data calculation part is fast but for technical reasons this is very slow, (Allocation of contigous memory streams, copying, adding etc). Even better, when you are really sloppy and work with 7 days of customer data System.OutOfMemoryException : ). So what to do? Still blinded by the R.A.D. issue I continued ...
LinkedList items = ...
Golden, no longer an out of memory issue for n "Luke, pretend this is C++ and you are using a struct and a pointer offset" "Ben, I can't C# doesn't allow safe casting" "Use the force" So I map
Piece-wise memory allocation is painful. For a particular project analyzing end-of-life failure modes on satellite switch rings, I have a function that "simply" spins up a list (several million members) of "random" (but balanced) failures to test. It takes minutes to generate the list, most of this is memory allocations going on behind the scenes of the list management. I never bothered to "un-safe" the algorithm. :) And yes, zip is painfully slow. And when I looked at using a serialized zip stream, I discovered it's not a true stream - you can't just shove data in and expect it to stream it out as it works through the compression - the whole damn thing has to be compressed in memory first before it starts spitting out the compressed data to the stream. I discovered that years ago, I think I even wrote an article about it. Marc
-
Piece-wise memory allocation is painful. For a particular project analyzing end-of-life failure modes on satellite switch rings, I have a function that "simply" spins up a list (several million members) of "random" (but balanced) failures to test. It takes minutes to generate the list, most of this is memory allocations going on behind the scenes of the list management. I never bothered to "un-safe" the algorithm. :) And yes, zip is painfully slow. And when I looked at using a serialized zip stream, I discovered it's not a true stream - you can't just shove data in and expect it to stream it out as it works through the compression - the whole damn thing has to be compressed in memory first before it starts spitting out the compressed data to the stream. I discovered that years ago, I think I even wrote an article about it. Marc
For me the most painful lesson with the GZip was trying to zip multiple items, separately, to one stream. That was a mistake! Seems a lot of people have it too.
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. "And they, since they Were not the one dead, turned to their affairs" -- Robert Frost "All users always want Excel" --Ennis Lynch
-
Preface ... I knew better but I made the mistake anyway and I just felt like sharing, you probably don't want to read : ) My current mission is to author applications that have never been written before, ever, rapidly, for the power industry on "Massive Amounts" of data for users that want it yesterday but will change their minds constantly. So pretend you want to know how long a users power was out (and I know ahead of time the rules will change on me)
public class Foo{
public DateTime PowerOutTime;
public DateTime PowerOnTime;
public TimeSpan FigureOutJustHowGDLongThePowerWasOut;
public string CustomerId;
}Populate the list with a few hundred million records, do some work. Output, slow, of course, data-loading takes a long time, up to 60 to 100 seconds for a day. Ignore the fact that I can't do this work in the DB (there are reasons) First naive mistake, Let's put it in a generic List
List items = ...
Sure the data calculation part is fast but for technical reasons this is very slow, (Allocation of contigous memory streams, copying, adding etc). Even better, when you are really sloppy and work with 7 days of customer data System.OutOfMemoryException : ). So what to do? Still blinded by the R.A.D. issue I continued ...
LinkedList items = ...
Golden, no longer an out of memory issue for n "Luke, pretend this is C++ and you are using a struct and a pointer offset" "Ben, I can't C# doesn't allow safe casting" "Use the force" So I map
Keep working on it, Ennis! This industry is pathetic. I've tried for months to develop a report based on hourly meter readings from our supplier, delivered daily to my mailbox, just to generate a meaningful value for system losses. It turns out that it's impossible. Even the people who use this telemetry data to generate our monthly power bills tell me that it's useless to look at these values, because they can't trust them, and each meter reports a different number of digits' resolution. They actually still rely on monthly manual meter reads to generate our bill! This from fully automated generation and transmission facilities owned and operated by US Government agencies!!!
Will Rogers never met me.
-
Keep working on it, Ennis! This industry is pathetic. I've tried for months to develop a report based on hourly meter readings from our supplier, delivered daily to my mailbox, just to generate a meaningful value for system losses. It turns out that it's impossible. Even the people who use this telemetry data to generate our monthly power bills tell me that it's useless to look at these values, because they can't trust them, and each meter reports a different number of digits' resolution. They actually still rely on monthly manual meter reads to generate our bill! This from fully automated generation and transmission facilities owned and operated by US Government agencies!!!
Will Rogers never met me.
How about, Just assume .9 power factor for all residential meters and give me the KVA for yesterday but only have KW/h at the hour and Voltage at the hour? And, your not allowed to have an index because it takes up too much space :)
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. "And they, since they Were not the one dead, turned to their affairs" -- Robert Frost "All users always want Excel" --Ennis Lynch
-
How about, Just assume .9 power factor for all residential meters and give me the KVA for yesterday but only have KW/h at the hour and Voltage at the hour? And, your not allowed to have an index because it takes up too much space :)
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. "And they, since they Were not the one dead, turned to their affairs" -- Robert Frost "All users always want Excel" --Ennis Lynch
:laugh: :laugh: It doesn't seem that your customer understands power at all; KVA is an instantaneous measure = kW/PF, and kW/h is a meaningless term. I pity you... :sigh:
Will Rogers never met me.
-
:laugh: :laugh: It doesn't seem that your customer understands power at all; KVA is an instantaneous measure = kW/PF, and kW/h is a meaningless term. I pity you... :sigh:
Will Rogers never met me.
Well, to be honest, everyone involved including myself understands the difference (which is surprising because I am bad at math). Unfortunately, when you have millions of meters all getting their readings at the hourly level we have to make guesses. I was more surprised that using a constant for power factor was OK, I could understand using a "best guess" based on available data KVA. Don't pity me too much we can actually bill from our data :p
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. "And they, since they Were not the one dead, turned to their affairs" -- Robert Frost "All users always want Excel" --Ennis Lynch
-
Well, to be honest, everyone involved including myself understands the difference (which is surprising because I am bad at math). Unfortunately, when you have millions of meters all getting their readings at the hourly level we have to make guesses. I was more surprised that using a constant for power factor was OK, I could understand using a "best guess" based on available data KVA. Don't pity me too much we can actually bill from our data :p
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. "And they, since they Were not the one dead, turned to their affairs" -- Robert Frost "All users always want Excel" --Ennis Lynch
Ennis Ray Lynch, Jr. wrote:
we can actually bill from our data
:omg: :wtf:
If your actions inspire others to dream more, learn more, do more and become more, you are a leader.-John Q. Adams
You must accept one of two basic premises: Either we are alone in the universe, or we are not alone in the universe. And either way, the implications are staggering.-Wernher von Braun
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.-Albert Einstein -
Well, to be honest, everyone involved including myself understands the difference (which is surprising because I am bad at math). Unfortunately, when you have millions of meters all getting their readings at the hourly level we have to make guesses. I was more surprised that using a constant for power factor was OK, I could understand using a "best guess" based on available data KVA. Don't pity me too much we can actually bill from our data :p
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. "And they, since they Were not the one dead, turned to their affairs" -- Robert Frost "All users always want Excel" --Ennis Lynch
Yeah, power factors don't change much, except in areas dominated by motors and other highly inductive loads that are switched on and off at various times. I have one substation that has a power factor for the entire service area of 1.000 - consistently! It's almost entirely residential, with a few well pumps, but there's also a 1.5 MG/D sewer plant on the line. How we pull that off is a mystery - just a lucky circumstance that the predominantly capacitive power lines exactly balance the inductive loads. I couldn't design it that way! :-D For residential, though, 0.9 is awfully low. Even in summer, when we have summer air conditioning loads that double the usual lighting loads of winter, we run a PF of about 0.95. The rest of the year it's 0.97 to 0.99.
Will Rogers never met me.
-
Preface ... I knew better but I made the mistake anyway and I just felt like sharing, you probably don't want to read : ) My current mission is to author applications that have never been written before, ever, rapidly, for the power industry on "Massive Amounts" of data for users that want it yesterday but will change their minds constantly. So pretend you want to know how long a users power was out (and I know ahead of time the rules will change on me)
public class Foo{
public DateTime PowerOutTime;
public DateTime PowerOnTime;
public TimeSpan FigureOutJustHowGDLongThePowerWasOut;
public string CustomerId;
}Populate the list with a few hundred million records, do some work. Output, slow, of course, data-loading takes a long time, up to 60 to 100 seconds for a day. Ignore the fact that I can't do this work in the DB (there are reasons) First naive mistake, Let's put it in a generic List
List items = ...
Sure the data calculation part is fast but for technical reasons this is very slow, (Allocation of contigous memory streams, copying, adding etc). Even better, when you are really sloppy and work with 7 days of customer data System.OutOfMemoryException : ). So what to do? Still blinded by the R.A.D. issue I continued ...
LinkedList items = ...
Golden, no longer an out of memory issue for n "Luke, pretend this is C++ and you are using a struct and a pointer offset" "Ben, I can't C# doesn't allow safe casting" "Use the force" So I map
RAD = Rapid Application Development This is not the same as the Development of Rapid Applications!
I'll be more enthusiastic about encouraging thinking outside the box when there's evidence of any thinking going on inside it. - pTerry
sawilde @ GitHub