Back To Pretending I Know How To Program
-
Oh goody! Tonight I re-opened a project I started years ago, only to discover that I've forgotten more than I ever learned about C# in VS2010. Sheesh! Even though I've never been very good at it, I can't believe how much I've forgotten already; it's only been a couple of years since I tackled this problem... :sigh: Of course, since the last time I opened this project, Microsoft has completely changed ADO.Net, and added a bunch of crap I don't need or want. But it's just possible that my project might benefit from the changes, if only I had unlimited free time to study it and figure out what works. What hasn't changed, apparently, is the fact that it's really hard to create a useful program using technology that is constantly changing when you have about an hour a day to do it. And that only late at night after a long day at work doing something entirely unrelated to programming. My challenge: Produce a monthly energy report that allows us to track the losses in the transmission system and the distribution system for our utility. My payoff: A raise to a salary that is only $12K less than average for an engineer with a third of my experience, instead of $24k less. The problem is that, while we receive monthly reports on what we bought from our suppliers, we have no way to check them. We have meters of our own, but we don't read them on the same schedule as the suppliers. But I do get a daily email with an attached .csv report, by hour, of the energy received by our company, and I can use that to match up the external reports with our internal records. The trouble is that it's a massive effort to do by hand, so I'm trying to automate it. My plan is to read the .csv files, transmogrify them into datatable records, and store them in a SQL Server database for further analysis. Last time I got as far as writing a routine that reads the files, culls out the header and fluff lines using regular expressions, and splits the csv fields into discrete values. Then I bumped into the new ADO.Net stuff and got lost. Since then, ADO has changed yet again, and I'm even more lost and confused than before. Those of you who have helped me in the past, be patient; I've had so little time to devote to programming that virtually everything you've told me in the past is gone. I'll be haunting the forums soon, and asking the same dumb questions again; be patient, please... I'm old and slow, but still trainable. :)
Will Rogers never met me.
"My payoff: A raise to a salary that is only $12K less than average for an engineer with a third of my experience, instead of $24k less." D: A 12k pay bump? Man, I make 20k less than the low average for entry level programming. And I'm the team Lead! Wish I could do a project for a 12k bump, but it's kinda hard with a startup. XD
-
First, having written a couple of small LOB apps that I got away with naming "[somename] Transmogrifier", it's nice to see someone else use "transmogrify" correctly in a sentence. Second, you might want to look at avoiding the overhead of Visual Studio and recompiling for every change by using Powershell instead -- it read CSV files as datatables, or you can use regular expressions to read the files line by line (been there, done that with my own VB.NET Transmogrifiers).
-
First, having written a couple of small LOB apps that I got away with naming "[somename] Transmogrifier", it's nice to see someone else use "transmogrify" correctly in a sentence. Second, you might want to look at avoiding the overhead of Visual Studio and recompiling for every change by using Powershell instead -- it read CSV files as datatables, or you can use regular expressions to read the files line by line (been there, done that with my own VB.NET Transmogrifiers).
Hmmmm.... PowerShell has been around a while, but I have no idea what it does, what it's good for, or where to find out. Maybe it's time to Google for it. :-O
Will Rogers never met me.
-
"My payoff: A raise to a salary that is only $12K less than average for an engineer with a third of my experience, instead of $24k less." D: A 12k pay bump? Man, I make 20k less than the low average for entry level programming. And I'm the team Lead! Wish I could do a project for a 12k bump, but it's kinda hard with a startup. XD
Well, that's what was promised, though it would have been nice to get to "average" for once. But I'm the one who, in another set of circumstances, pointed out that a small company like ours would not normally have two engineers on the payroll, and our GM is the other engineer. If I stick to my guns asking for market value, I may get it just long enough for them to decide they don't need me at all. Yes, it would be stupid of the Board to act in such a way, as the GM already spends 110% of his available time doing management stuff and politics. But when has an appointed Board ever acted rationally? I'll take what I can get and be happy with it! :-D
Will Rogers never met me.
-
Hmmmm.... PowerShell has been around a while, but I have no idea what it does, what it's good for, or where to find out. Maybe it's time to Google for it. :-O
Will Rogers never met me.
-
Oh goody! Tonight I re-opened a project I started years ago, only to discover that I've forgotten more than I ever learned about C# in VS2010. Sheesh! Even though I've never been very good at it, I can't believe how much I've forgotten already; it's only been a couple of years since I tackled this problem... :sigh: Of course, since the last time I opened this project, Microsoft has completely changed ADO.Net, and added a bunch of crap I don't need or want. But it's just possible that my project might benefit from the changes, if only I had unlimited free time to study it and figure out what works. What hasn't changed, apparently, is the fact that it's really hard to create a useful program using technology that is constantly changing when you have about an hour a day to do it. And that only late at night after a long day at work doing something entirely unrelated to programming. My challenge: Produce a monthly energy report that allows us to track the losses in the transmission system and the distribution system for our utility. My payoff: A raise to a salary that is only $12K less than average for an engineer with a third of my experience, instead of $24k less. The problem is that, while we receive monthly reports on what we bought from our suppliers, we have no way to check them. We have meters of our own, but we don't read them on the same schedule as the suppliers. But I do get a daily email with an attached .csv report, by hour, of the energy received by our company, and I can use that to match up the external reports with our internal records. The trouble is that it's a massive effort to do by hand, so I'm trying to automate it. My plan is to read the .csv files, transmogrify them into datatable records, and store them in a SQL Server database for further analysis. Last time I got as far as writing a routine that reads the files, culls out the header and fluff lines using regular expressions, and splits the csv fields into discrete values. Then I bumped into the new ADO.Net stuff and got lost. Since then, ADO has changed yet again, and I'm even more lost and confused than before. Those of you who have helped me in the past, be patient; I've had so little time to devote to programming that virtually everything you've told me in the past is gone. I'll be haunting the forums soon, and asking the same dumb questions again; be patient, please... I'm old and slow, but still trainable. :)
Will Rogers never met me.
-
Well, that's what was promised, though it would have been nice to get to "average" for once. But I'm the one who, in another set of circumstances, pointed out that a small company like ours would not normally have two engineers on the payroll, and our GM is the other engineer. If I stick to my guns asking for market value, I may get it just long enough for them to decide they don't need me at all. Yes, it would be stupid of the Board to act in such a way, as the GM already spends 110% of his available time doing management stuff and politics. But when has an appointed Board ever acted rationally? I'll take what I can get and be happy with it! :-D
Will Rogers never met me.
I read this one with interest , I have very similar issues with my work. In my case it is machine shop software, and python3, .net & C# for hardware. I think that there have been some really good suggestions here and a lot of empathy for your situation. I was pretty well in the danger zone in my our startup, keeping the machine shop on track and trying to squeeze out any extra time for coding. What saved me was figuring out that the board was responsive to well prepared cases where I broke the coding out into chunks and subcontracted some of them. In my case it supposedly "changed their perception of me from a grease monkey, to a competent manager". I'm still employed in any case and three months ago I was on the fast track to trouble. Anyway I wish you luck. There are lots of ways to solve any problem.
-
Oh goody! Tonight I re-opened a project I started years ago, only to discover that I've forgotten more than I ever learned about C# in VS2010. Sheesh! Even though I've never been very good at it, I can't believe how much I've forgotten already; it's only been a couple of years since I tackled this problem... :sigh: Of course, since the last time I opened this project, Microsoft has completely changed ADO.Net, and added a bunch of crap I don't need or want. But it's just possible that my project might benefit from the changes, if only I had unlimited free time to study it and figure out what works. What hasn't changed, apparently, is the fact that it's really hard to create a useful program using technology that is constantly changing when you have about an hour a day to do it. And that only late at night after a long day at work doing something entirely unrelated to programming. My challenge: Produce a monthly energy report that allows us to track the losses in the transmission system and the distribution system for our utility. My payoff: A raise to a salary that is only $12K less than average for an engineer with a third of my experience, instead of $24k less. The problem is that, while we receive monthly reports on what we bought from our suppliers, we have no way to check them. We have meters of our own, but we don't read them on the same schedule as the suppliers. But I do get a daily email with an attached .csv report, by hour, of the energy received by our company, and I can use that to match up the external reports with our internal records. The trouble is that it's a massive effort to do by hand, so I'm trying to automate it. My plan is to read the .csv files, transmogrify them into datatable records, and store them in a SQL Server database for further analysis. Last time I got as far as writing a routine that reads the files, culls out the header and fluff lines using regular expressions, and splits the csv fields into discrete values. Then I bumped into the new ADO.Net stuff and got lost. Since then, ADO has changed yet again, and I'm even more lost and confused than before. Those of you who have helped me in the past, be patient; I've had so little time to devote to programming that virtually everything you've told me in the past is gone. I'll be haunting the forums soon, and asking the same dumb questions again; be patient, please... I'm old and slow, but still trainable. :)
Will Rogers never met me.
Why not just use an old version of the framework? Unless the point of the project is to learn the new technology, you'd be able to keep making progress on the task itself without having to constantly retrain.
We can program with only 1's, but if all you've got are zeros, you've got nothing.
-
Oh goody! Tonight I re-opened a project I started years ago, only to discover that I've forgotten more than I ever learned about C# in VS2010. Sheesh! Even though I've never been very good at it, I can't believe how much I've forgotten already; it's only been a couple of years since I tackled this problem... :sigh: Of course, since the last time I opened this project, Microsoft has completely changed ADO.Net, and added a bunch of crap I don't need or want. But it's just possible that my project might benefit from the changes, if only I had unlimited free time to study it and figure out what works. What hasn't changed, apparently, is the fact that it's really hard to create a useful program using technology that is constantly changing when you have about an hour a day to do it. And that only late at night after a long day at work doing something entirely unrelated to programming. My challenge: Produce a monthly energy report that allows us to track the losses in the transmission system and the distribution system for our utility. My payoff: A raise to a salary that is only $12K less than average for an engineer with a third of my experience, instead of $24k less. The problem is that, while we receive monthly reports on what we bought from our suppliers, we have no way to check them. We have meters of our own, but we don't read them on the same schedule as the suppliers. But I do get a daily email with an attached .csv report, by hour, of the energy received by our company, and I can use that to match up the external reports with our internal records. The trouble is that it's a massive effort to do by hand, so I'm trying to automate it. My plan is to read the .csv files, transmogrify them into datatable records, and store them in a SQL Server database for further analysis. Last time I got as far as writing a routine that reads the files, culls out the header and fluff lines using regular expressions, and splits the csv fields into discrete values. Then I bumped into the new ADO.Net stuff and got lost. Since then, ADO has changed yet again, and I'm even more lost and confused than before. Those of you who have helped me in the past, be patient; I've had so little time to devote to programming that virtually everything you've told me in the past is gone. I'll be haunting the forums soon, and asking the same dumb questions again; be patient, please... I'm old and slow, but still trainable. :)
Will Rogers never met me.
I love it! That happens to me with C++ -- not to mention the fact that every time I re-encounter it, the platforms, operating systems and general milieu in which I'm programming has changed so radically that it's like starting all over again. I feel your pain...!!
-
Oh goody! Tonight I re-opened a project I started years ago, only to discover that I've forgotten more than I ever learned about C# in VS2010. Sheesh! Even though I've never been very good at it, I can't believe how much I've forgotten already; it's only been a couple of years since I tackled this problem... :sigh: Of course, since the last time I opened this project, Microsoft has completely changed ADO.Net, and added a bunch of crap I don't need or want. But it's just possible that my project might benefit from the changes, if only I had unlimited free time to study it and figure out what works. What hasn't changed, apparently, is the fact that it's really hard to create a useful program using technology that is constantly changing when you have about an hour a day to do it. And that only late at night after a long day at work doing something entirely unrelated to programming. My challenge: Produce a monthly energy report that allows us to track the losses in the transmission system and the distribution system for our utility. My payoff: A raise to a salary that is only $12K less than average for an engineer with a third of my experience, instead of $24k less. The problem is that, while we receive monthly reports on what we bought from our suppliers, we have no way to check them. We have meters of our own, but we don't read them on the same schedule as the suppliers. But I do get a daily email with an attached .csv report, by hour, of the energy received by our company, and I can use that to match up the external reports with our internal records. The trouble is that it's a massive effort to do by hand, so I'm trying to automate it. My plan is to read the .csv files, transmogrify them into datatable records, and store them in a SQL Server database for further analysis. Last time I got as far as writing a routine that reads the files, culls out the header and fluff lines using regular expressions, and splits the csv fields into discrete values. Then I bumped into the new ADO.Net stuff and got lost. Since then, ADO has changed yet again, and I'm even more lost and confused than before. Those of you who have helped me in the past, be patient; I've had so little time to devote to programming that virtually everything you've told me in the past is gone. I'll be haunting the forums soon, and asking the same dumb questions again; be patient, please... I'm old and slow, but still trainable. :)
Will Rogers never met me.
I guess SQL Express doesn't come with Integration Services (SSIS)? I'm starting to use SSIS for my ETL projects - it's pretty well featured and there's a lit of examples, samples, and help out there.
-
I guess SQL Express doesn't come with Integration Services (SSIS)? I'm starting to use SSIS for my ETL projects - it's pretty well featured and there's a lit of examples, samples, and help out there.
Personally, I'm heading away from SSIS in favor of Powershell for increased flexibility and vastly increased debugging transparency.
-
I love it! That happens to me with C++ -- not to mention the fact that every time I re-encounter it, the platforms, operating systems and general milieu in which I'm programming has changed so radically that it's like starting all over again. I feel your pain...!!
Dan Sutton wrote:
I love it! That happens to me with C++ -- not to mention...
I read that as "That happens to me with C plus plus minus minus". I had a brief near heart attack wondering what horrible new extension the C++ committee was foisting on us this time... :doh:
// ToDo: // Put Signature Here
-
Dan Sutton wrote:
I love it! That happens to me with C++ -- not to mention...
I read that as "That happens to me with C plus plus minus minus". I had a brief near heart attack wondering what horrible new extension the C++ committee was foisting on us this time... :doh:
// ToDo: // Put Signature Here
Brilliant!
-
Hehehe... Two subtleties, actually. Number one is that I don't care to use Access anymore, so I'm targeting SQL Server (Express, of course) this time around. The second is not a subtlety per se, just something I neglected to describe well. The csv files contain several lines of fluff in a format that changes from time to time, such as:
Date Range = Yesterday (09/01/2010)
Time Zone = MST
Sign Convention = GEN + LOAD + OTHER +
Include Zero = Yes
Execution Time: 09/02/2010 07:00 CST"Zone = RMS / AMPS"
"Date","Meter","HE1","HE2","HE3","HE4","HE5","HE6","HE7","HE8","HE9","HE10","HE11","HE12","HE13","HE14","HE15","HE16","HE17","HE18","HE19","HE20","HE21","HE22","HE23","HE24","On Peak","Off Peak","Total"
The data looks like: "09/01/2010","BCI891 Out","0.000","0.000","0.000","0.000","0.000","0.000","4.000","4.000","4.000","0.000","4.000","4.000","4.000","4.000","4.000","0.000","4.000","0.000","5.000","0.000","0.000","4.000","0.000","0.000","45.000","0.000","45.000" "","DAD348 Out","0.000","0.000","0.000","0.000","0.000","0.000","0.000","2.975","3.857","4.284","4.550","4.879","5.103","5.390","5.544","5.712","5.824","5.628","5.502","5.523","5.460","5.033","4.634","4.382","75.264","9.016","84.280" "","NON921A Out","-6.200","-5.800","-5.600","-5.300","-5.400","-5.500","-5.300","-5.800","-6.500","-7.300","-8.000","-9.100","-10.100","-10.900","-11.500","-11.700","-11.900","-11.700","-11.100","-10.700","-10.100","-9.000","-8.100","-7.300","-150.700","-49.200","-199.900" "","TOP928 In","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000","0.000" "","TOP928 Out","7.490","6.940","6.740","6.440","6.430","6.570","6.400","7.070","7.870","8.710","9.650","10.850","12.000","13.020","13.730","14.080","14.120","14.010","13.210","12.670","11.850","10.680","9.620","8.750","179.920","58.980","238.900" "09/01/2010","","1.290","1.140","1.140","1.140","1.030","1.070","5.100","8.245","9.227","5.694","10.200","10.629","11.003","11.510","11.774","8.092","12.044","7.938","12.612","7.493","7.210","10.713","6.154","5.832","149.484","18.796","168.280" The fluff and header stuff at the beginning, along with any blank lines, I discard. I also dump the last data line, since it's just a column total and shouldn't be preserved in the database. So I probably could have described it better, but I wasn't asking a question here, of course. My
this Kind of Thing is my daily bread and butter. is the size of the Header(fluff) always the same Count of lines? If so, use a StreamReader, Count the lines you're reading until you reach the data. As I understand it, every line of the data represents a different Meter with its own data, the strucure is always the same with the same count of elements and the same Positions. If so, read the lines of Data into an Array of string. from then on you can do with the seperate fields as you like as Long as you know the Position, for example date is alway on the first Position, in a Array this means it is on Position '0'.
private void readLines(string inputFile, int headerLineCount)
{
List inputList = new List();Encoding coding = Encoding.GetEncoding(1252); //Ansi-Encoding, Change to 850 for ASCII, 1200 for Unicode UTF16, 65001 for UTF8 etc. StreamReader reader = new StreamReader(inputFile, coding); int lineIndex = 0; while (reader.EndOfStream != true)//Read until the file has no more lines to read. { string inputLine = reader.ReadLine();//read the data here if (lineIndex >= headerLineCount)//This Code is executed only after the set amount of lines (header) has been overread. { string\[\] inputElements = inputLine.Replace("\\"", "").Split(','); //We replace the '"' with an empty string and split the line into seperate Elements. You can use any char as a delemiter, for excample ';' or '!' etc. meterInput newInput = new meterInput(); newInput.Date = Convert.ToDateTime(inputElements\[0\]);// Here we use the first element of the 'inputElements' and convert it to a Datetime. newInput.Name = inputElements\[1\]; newInput.HE1 = Convert.ToDecimal(inputElements\[2\]); newInput.HE2 = Convert.ToDecimal(inputElements\[3\]); newInput.HE3 = Convert.ToDecimal(inputElements\[4\]); newInput.HE4 = Convert.ToDecimal(inputElements\[5\]); //... Add the rest of the data. newInput.OnPeak = Convert.ToDecimal(inputElements\[26\]); newInput.OffPeak = Convert.ToDecimal(inputElements\[27\]); newInput.Total = Convert.ToDecimal(inputElements\[28\]); inputList.Add(newInput); // Add the complete element to -
this Kind of Thing is my daily bread and butter. is the size of the Header(fluff) always the same Count of lines? If so, use a StreamReader, Count the lines you're reading until you reach the data. As I understand it, every line of the data represents a different Meter with its own data, the strucure is always the same with the same count of elements and the same Positions. If so, read the lines of Data into an Array of string. from then on you can do with the seperate fields as you like as Long as you know the Position, for example date is alway on the first Position, in a Array this means it is on Position '0'.
private void readLines(string inputFile, int headerLineCount)
{
List inputList = new List();Encoding coding = Encoding.GetEncoding(1252); //Ansi-Encoding, Change to 850 for ASCII, 1200 for Unicode UTF16, 65001 for UTF8 etc. StreamReader reader = new StreamReader(inputFile, coding); int lineIndex = 0; while (reader.EndOfStream != true)//Read until the file has no more lines to read. { string inputLine = reader.ReadLine();//read the data here if (lineIndex >= headerLineCount)//This Code is executed only after the set amount of lines (header) has been overread. { string\[\] inputElements = inputLine.Replace("\\"", "").Split(','); //We replace the '"' with an empty string and split the line into seperate Elements. You can use any char as a delemiter, for excample ';' or '!' etc. meterInput newInput = new meterInput(); newInput.Date = Convert.ToDateTime(inputElements\[0\]);// Here we use the first element of the 'inputElements' and convert it to a Datetime. newInput.Name = inputElements\[1\]; newInput.HE1 = Convert.ToDecimal(inputElements\[2\]); newInput.HE2 = Convert.ToDecimal(inputElements\[3\]); newInput.HE3 = Convert.ToDecimal(inputElements\[4\]); newInput.HE4 = Convert.ToDecimal(inputElements\[5\]); //... Add the rest of the data. newInput.OnPeak = Convert.ToDecimal(inputElements\[26\]); newInput.OffPeak = Convert.ToDecimal(inputElements\[27\]); newInput.Total = Convert.ToDecimal(inputElements\[28\]); inputList.Add(newInput); // Add the complete element toThanks, Peter! This is about where I'd got to last time, but your way is much cleaner. I can learn from this, which is why I enjoy doing it. :-D The # of lines of header varies, though, and only the first and last lines of meter data includes the date, so there's a little modification to be done, but that's easy stuff.
Will Rogers never met me.