Did you know...
-
John Simmons / outlaw programmer wrote:
I wish people here (not you but some others) would stop f*ckin assuming I'm a rookie programmer.
Just like at work, other people mess up, but you get the blame!
John Simmons / outlaw programmer wrote:
I have more years in the industry than most people on CP have even been alive.
That's no guarantee for actually being a good programmer. For example, the programmer who gives you 3.5 GB of XML in a single file probably says the same :rolleyes:
Best, Sander sanderrossel.com Continuous Integration, Delivery, and Deployment arrgh.js - Bringing LINQ to JavaScript Object-Oriented Programming in C# Succinctly
Sander Rossel wrote:
For example, the programmer who gives you 3.5 GB of XML in a single file probably says the same
We don't get the files from programmers - we get them from security nazis.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
Methinks you need to rethink your XML design
CQ de W5ALT
Walt Fair, Jr., P. E. Comport Computing Specializing in Technical Engineering Software
It ain't my design, and it won't be changing to anything better.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
Sander Rossel wrote:
For example, the programmer who gives you 3.5 GB of XML in a single file probably says the same
We don't get the files from programmers - we get them from security nazis.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013John Simmons / outlaw programmer wrote:
we get them from security nazis.
And on the other side of that barrier there is some poor sod producing the xml. Or it was designed in the 90s and they refuse to even consider changing something that works - sort of.
Never underestimate the power of human stupidity - RAH I'm old. I know stuff - JSOP
-
John Simmons / outlaw programmer wrote:
we get them from security nazis.
And on the other side of that barrier there is some poor sod producing the xml. Or it was designed in the 90s and they refuse to even consider changing something that works - sort of.
Never underestimate the power of human stupidity - RAH I'm old. I know stuff - JSOP
a scan tool called Nessus generates the file. I know nothing about it, or it’s configurability where file generation is concerned.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
John Simmons / outlaw programmer wrote:
we get them from security nazis.
And on the other side of that barrier there is some poor sod producing the xml. Or it was designed in the 90s and they refuse to even consider changing something that works - sort of.
Never underestimate the power of human stupidity - RAH I'm old. I know stuff - JSOP
More likely it was designed in the 90's when the log data was small and (the then new and cutting edge) XML made some sense. But ... the developer who wrote that moved on, and file formats are boring, so the new guy just tested it worked in small scale and worked on the sexier stuff. And now ... intrusion / vulnerability data has grown like everything else and it's just a silly decision with hindsight.
Sent from my Amstrad PC 1640 Never throw anything away, Griff Bad command or file name. Bad, bad command! Sit! Stay! Staaaay... AntiTwitter: @DalekDave is now a follower!
-
The 6GB XML file I have to read is the backup of a third-party system. It's not only XML, but it's all name/value pairs.
-
Combining XmlReader and LinqToXML, the memory consumption never goes above 350mb, and it takes about 45 minutes to run though the sample files (this includes adding the data to the database, one record at a time (426,000 records). When I add a dash of TPL, it only takes about 9 minutes to process the same three files. I think I could get it even faster if I inserted multiple records per query, but I'm tired of dickin' with it.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013Well, for the most part I'm limited to built-in SSIS components. Potentially I could write something custom, as I have for JSON and CSV files (ones which aren't stable enough for the flat-file components).
-
The 6GB XML file I have to read is the backup of a third-party system. It's not only XML, but it's all name/value pairs.
-
Don't tell me. I mostly use it for configuration.
-
OriginalGriff wrote:
Have these people never heard of databases?
That's our job. :) Got memory consumption down to no more than 350mb and it only takes 9 minutes to process my three sample files, for a total of 426,000 records. I'm going to look awesome on Tuesday. Upside, this app replaces a large perl script that was doing the same job, and everyone in the shop can maintain it because - well - it's not perl. :)
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013John Simmons / outlaw programmer wrote:
I'm going to look awesome on Tuesday.
That only if the other morons people appreciate your work, not the first time awesome tools that are real improvements get dumped because a couple of idiots co-workers say: - We have always done it this way - That is not going to work (without even giving a try) - Or similar crap arguments... and not even give a damned "Thank you"
M.D.V. ;) If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about? Help me to understand what I'm saying, and I'll explain it better to you Rating helpful answers is nice, but saying thanks can be even nicer.
-
John Simmons / outlaw programmer wrote:
I'm going to look awesome on Tuesday.
That only if the other morons people appreciate your work, not the first time awesome tools that are real improvements get dumped because a couple of idiots co-workers say: - We have always done it this way - That is not going to work (without even giving a try) - Or similar crap arguments... and not even give a damned "Thank you"
M.D.V. ;) If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about? Help me to understand what I'm saying, and I'll explain it better to you Rating helpful answers is nice, but saying thanks can be even nicer.
I've already gotten thank-yous for this. They're grateful that they don't have to maintain that monster perl script anymore.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
More likely it was designed in the 90's when the log data was small and (the then new and cutting edge) XML made some sense. But ... the developer who wrote that moved on, and file formats are boring, so the new guy just tested it worked in small scale and worked on the sexier stuff. And now ... intrusion / vulnerability data has grown like everything else and it's just a silly decision with hindsight.
Sent from my Amstrad PC 1640 Never throw anything away, Griff Bad command or file name. Bad, bad command! Sit! Stay! Staaaay... AntiTwitter: @DalekDave is now a follower!
I'm sure they're suffering from the same thing we all have to deal with - management that doesn't (want to) see a reason to re-architect the app that generates the files.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
VS(2017) has a maximum supported file size of 10mb? I just found out myself while trying to load a 925mb xml file.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013Did you know you can increase that maximum, if you're feeling brave? :) registry - Large XML Files in VS 2017 15.1 - Stack Overflow[^] No idea whether you can increase it enough for such a ludicrously huge file, though.
"These people looked deep within my soul and assigned me a number based on the order in which I joined." - Homer
-
Did you know you can increase that maximum, if you're feeling brave? :) registry - Large XML Files in VS 2017 15.1 - Stack Overflow[^] No idea whether you can increase it enough for such a ludicrously huge file, though.
"These people looked deep within my soul and assigned me a number based on the order in which I joined." - Homer
yeah, i knew that, and i imaging that if its an integer value, int.maxvalue would be reasonable.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
