Did you know, final
-
I started with multi-gb size XML files that would cause out-of-memory exceptions. After I resolved that issue (by using
XmlTextReader/XElementinstead ofXDocument), it was taking over 20 minutes to process the sample file set (434,000 rows). This is where I was on Friday. I added some TPL love and got it down to about 12 minutes. I reworked the target tables. At first, I was creating a single monolithic flat table, and a lot of data was being repeated for every record. I created two tables (one that holds the repeating data, and one that holds the unique data), and got it down to just a little over 6 minutes. This code will replace a Perl script that: 0) Creates a bunch of smaller XML files, probably so it doesn't run out of memory 1) Took over an hour to process the same set of sample files. 2) Did not return all of the data (because the stakeholder doesn't really need it) Added benefits: 0) My code can process ZIP (extracts NESSUS files from them for processing) and bare naked NESSUS files 1) Processes multiple files with a single execution 2) Puts ALL of the data into the database Caveat: Because of the limited hardware the app will actually be running on, processing time will take longer (because of the TPL's default behavior), but not significantly so, and CERTAINLY not take an hour. It's been a good weekend. :)".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013And now ... the overtime claim ... :laugh:
Sent from my Amstrad PC 1640 Never throw anything away, Griff Bad command or file name. Bad, bad command! Sit! Stay! Staaaay... AntiTwitter: @DalekDave is now a follower!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt -
And now ... the overtime claim ... :laugh:
Sent from my Amstrad PC 1640 Never throw anything away, Griff Bad command or file name. Bad, bad command! Sit! Stay! Staaaay... AntiTwitter: @DalekDave is now a follower!
Nope, just did it for the fun of it.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
I started with multi-gb size XML files that would cause out-of-memory exceptions. After I resolved that issue (by using
XmlTextReader/XElementinstead ofXDocument), it was taking over 20 minutes to process the sample file set (434,000 rows). This is where I was on Friday. I added some TPL love and got it down to about 12 minutes. I reworked the target tables. At first, I was creating a single monolithic flat table, and a lot of data was being repeated for every record. I created two tables (one that holds the repeating data, and one that holds the unique data), and got it down to just a little over 6 minutes. This code will replace a Perl script that: 0) Creates a bunch of smaller XML files, probably so it doesn't run out of memory 1) Took over an hour to process the same set of sample files. 2) Did not return all of the data (because the stakeholder doesn't really need it) Added benefits: 0) My code can process ZIP (extracts NESSUS files from them for processing) and bare naked NESSUS files 1) Processes multiple files with a single execution 2) Puts ALL of the data into the database Caveat: Because of the limited hardware the app will actually be running on, processing time will take longer (because of the TPL's default behavior), but not significantly so, and CERTAINLY not take an hour. It's been a good weekend. :)".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013And now the streaming SignalR version please :-\
-
I started with multi-gb size XML files that would cause out-of-memory exceptions. After I resolved that issue (by using
XmlTextReader/XElementinstead ofXDocument), it was taking over 20 minutes to process the sample file set (434,000 rows). This is where I was on Friday. I added some TPL love and got it down to about 12 minutes. I reworked the target tables. At first, I was creating a single monolithic flat table, and a lot of data was being repeated for every record. I created two tables (one that holds the repeating data, and one that holds the unique data), and got it down to just a little over 6 minutes. This code will replace a Perl script that: 0) Creates a bunch of smaller XML files, probably so it doesn't run out of memory 1) Took over an hour to process the same set of sample files. 2) Did not return all of the data (because the stakeholder doesn't really need it) Added benefits: 0) My code can process ZIP (extracts NESSUS files from them for processing) and bare naked NESSUS files 1) Processes multiple files with a single execution 2) Puts ALL of the data into the database Caveat: Because of the limited hardware the app will actually be running on, processing time will take longer (because of the TPL's default behavior), but not significantly so, and CERTAINLY not take an hour. It's been a good weekend. :)".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013Just don't get too pissed off if it's sidelined for political or egoistic reasons. That happens all the time, and (rightly) blowing your top only gives them more ammunition to snipe at you with.
I wanna be a eunuchs developer! Pass me a bread knife!
-
Just don't get too pissed off if it's sidelined for political or egoistic reasons. That happens all the time, and (rightly) blowing your top only gives them more ammunition to snipe at you with.
I wanna be a eunuchs developer! Pass me a bread knife!
Well, I was asked to do it if I had time, so... I don't really care if it's used or not. Pretty much everything I've ever written for the government is no longer used for the very reasons you listed. I've kinda GOTTEN used to it. I see it as a learning experience, especially due to the large XML file stuff I had to work around.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
Well, I was asked to do it if I had time, so... I don't really care if it's used or not. Pretty much everything I've ever written for the government is no longer used for the very reasons you listed. I've kinda GOTTEN used to it. I see it as a learning experience, especially due to the large XML file stuff I had to work around.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013That's a good perspective. You've done work to be proud of. If the @rseholes are too busy blowing up their reps to be proud of it, it's because they're just @rseholes.
I wanna be a eunuchs developer! Pass me a bread knife!
-
Well, I was asked to do it if I had time, so... I don't really care if it's used or not. Pretty much everything I've ever written for the government is no longer used for the very reasons you listed. I've kinda GOTTEN used to it. I see it as a learning experience, especially due to the large XML file stuff I had to work around.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
I think you have the basis of an article here John
We can’t stop here, this is bat country - Hunter S Thompson RIP
i might...
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
i might...
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013Please do! I was looking through the whole thread thinking how this would make a great article and hoping you would be inspired :)
<sig notetoself="think of a better signature"> <first>Jim</first> <last>Meadors</last> </sig>
-
Nope, just did it for the fun of it.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013Are you sure you didn't do it to implicitly say someone... I told you so? :rolleyes: :-D
M.D.V. ;) If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about? Help me to understand what I'm saying, and I'll explain it better to you Rating helpful answers is nice, but saying thanks can be even nicer.
-
Are you sure you didn't do it to implicitly say someone... I told you so? :rolleyes: :-D
M.D.V. ;) If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about? Help me to understand what I'm saying, and I'll explain it better to you Rating helpful answers is nice, but saying thanks can be even nicer.
Nope - there were several challenging aspects to the project, and I enjoy a decent programming challenge.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
Well, I was asked to do it if I had time, so... I don't really care if it's used or not. Pretty much everything I've ever written for the government is no longer used for the very reasons you listed. I've kinda GOTTEN used to it. I see it as a learning experience, especially due to the large XML file stuff I had to work around.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013John Simmons / outlaw programmer wrote:
everything I've ever written for the government is no longer used
That's the reason I'm glad I got out of defense contracting. Of the last two projects I did for the Air Force in he late 1980's, one took over ten man-years to develop and was used for two weeks. The second was a bit smaller, taking about 3 man-years, but was never used at all. The only thing worse than when they call your baby ugly is when they are indifferent to it.
Software Zen:
delete this; -
John Simmons / outlaw programmer wrote:
everything I've ever written for the government is no longer used
That's the reason I'm glad I got out of defense contracting. Of the last two projects I did for the Air Force in he late 1980's, one took over ten man-years to develop and was used for two weeks. The second was a bit smaller, taking about 3 man-years, but was never used at all. The only thing worse than when they call your baby ugly is when they are indifferent to it.
Software Zen:
delete this;I simply don't care anymore - I was employed, and I did what they wanted. If they want to throw it all in the crapper, that's their perogative.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
Nope - there were several challenging aspects to the project, and I enjoy a decent programming challenge.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013When you're stuck maintaining an existing project, optimizing the living crap out of somebody else's inefficient design can be rather rewarding, even when it's only to make yourself feel smug. Not suggesting this is the case here. :-) Those time figures speak for themselves.
-
I simply don't care anymore - I was employed, and I did what they wanted. If they want to throw it all in the crapper, that's their perogative.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013John Simmons / outlaw programmer wrote:
I simply don't care anymore - I was employed, and I did what they wanted. If they want to throw it all in the crapper, that's their perogative.
The voice of the experienced software developer. Once said (by me) to a younger developer who was commiserating that none of the stuff he's working on will ever be remembered by anyone: "Sorry to disappoint you Michelangelo, but we're not building cathedrals."
