Subtle Bug in Subtle Bugs system
-
Ri Qen-Sin wrote:
all the e-mails
Depends which list. We were running at around 600,000 emails an hour on the daily Insider today. A new land-speed record for us but we've not really optimised our setup that much and there's still more scope for speed improvements. Our previous mailout system pushed 13,000 an hour. I love C#.
cheers, Chris Maunder
CodeProject.com : C++ MVP
How about the resource usage (memory/CPU) while your mailout service is running? As a coincidence, we had just a topic in Lounge sometime today (India) afternoon regarding a comparison of .NET GC and Java GC. The permalink is here: http://www.codeproject.com/script/Forums/View.aspx?fid=1159&msg=2452320[^]
Vasudevan Deepak Kumar Personal Homepage
Tech Gossips
A pessimist sees only the dark side of the clouds, and mopes; a philosopher sees both sides, and shrugs; an optimist doesn't see the clouds at all - he's walking on them. --Leonard Louis Levinson -
Ri Qen-Sin wrote:
all the e-mails
Depends which list. We were running at around 600,000 emails an hour on the daily Insider today. A new land-speed record for us but we've not really optimised our setup that much and there's still more scope for speed improvements. Our previous mailout system pushed 13,000 an hour. I love C#.
cheers, Chris Maunder
CodeProject.com : C++ MVP
Just wondering, did you send e-mail one recipient at a time or your recipient list is a mile and half long?
Chris Maunder wrote:
We were running at around 600,000 emails an hour on the daily Insider today.
This is almost as good as the Informercial King's claim of "sending e-mail to one million customers in zero second". :-D
My .NET Business Application Framework My Home Page
modified on Thursday, March 6, 2008 11:28 AM
-
Just wondering, did you send e-mail one recipient at a time or your recipient list is a mile and half long?
Chris Maunder wrote:
We were running at around 600,000 emails an hour on the daily Insider today.
This is almost as good as the Informercial King's claim of "sending e-mail to one million customers in zero second". :-D
My .NET Business Application Framework My Home Page
modified on Thursday, March 6, 2008 11:28 AM
Hi
-
This is called Irony. Our mailout system suddenly decided to go slow. Normally we can send the newsletter out at 200K - 300K emails an hour but today it was on a go-slow and was pushing 3,000 an hour. With nearly a million subscribers waiting to hear the news this wasn't looking good yet I had no idea what was going wrong. First the obvious things: 1. Check the database. Everything seemed OK, in that it was slow across our live and all our test servers. Some of these servers are still in the pre-2.2 code state so if it was a 2.2 code issue it would hopefully show up there. Nothing. 2. Check the code. Again, manual checks of all relevant code to see what the changes between 2.1 and 2.2 code were and nothing, apart from the known and tested changes, was showing up. 3. Check the mail servers. Even though I wasn't using any mail servers for the tests I was running I checked them. It was all fine 4. Threading. I disabled threading to see if that was the issue. It wasn't 5. Performance monitor: what was happening? It turns out that tens of thousands of objects were being created where before merely hundreds were. What the...? There was definitely extra processing going on. But where and why? 6. Step through everything. I was doing the previous tests on our staging server to simulate the live scenario so now I switched to the my home machine and had a look. It was fast. Lightning fast. Object creation was normal and everything perfect. Ooohkaaay.... 7. Rebuild and Redeploy. Ensuring I got the latest code from the source code repository I did a clean and full rebuild on my office machine (better connection to the hosting centre from there) and deployed. Retest and nothing. OK, this is weird. 8. Get desperate The version on my local machine at home works fast. The live version works slow. Try uploading the version from my home machine and not my office machine. It works! But they are from the same code base...? Now I'm scared. And then I saw it. A small file called 'ErrorLog.txt' that works as our last line of defence in error trapping was being created by the live version. The file contained thousands of messages about a missing config value, but the error was so deep it didn't even get the chance to tell me which value it was. Bring out WinMerge and check all config files and there we have it. A value in the config files was different Local version: SubtleBugsId Live version: SubtleBugId It turns out the live version used t
Please read the forum guidelines. This isn't an example of a subtle bug. It's an example of a mistake, an error or oversight. Sofware wasn't to blame....the programmer was! Just joking :laugh: Been down that path too. Venting frustration outside, yelling WTF and occasionally throwing objects against other harder objects. Then epiphany and :doh: :-\ Nice job tracking it down.
-
Damn, that's funny. By the way, how long does it take to send out all the e-mails?
So the creationist says: Everything must have a designer. God designed everything. I say: Why is God the only exception? Why not make the "designs" (like man) exceptions and make God a creation of man?
That's a real cool signature you have Ri. I just loved it :cool:
It's better to know some of the questions than all of the answers.
Pravin. -
This is called Irony. Our mailout system suddenly decided to go slow. Normally we can send the newsletter out at 200K - 300K emails an hour but today it was on a go-slow and was pushing 3,000 an hour. With nearly a million subscribers waiting to hear the news this wasn't looking good yet I had no idea what was going wrong. First the obvious things: 1. Check the database. Everything seemed OK, in that it was slow across our live and all our test servers. Some of these servers are still in the pre-2.2 code state so if it was a 2.2 code issue it would hopefully show up there. Nothing. 2. Check the code. Again, manual checks of all relevant code to see what the changes between 2.1 and 2.2 code were and nothing, apart from the known and tested changes, was showing up. 3. Check the mail servers. Even though I wasn't using any mail servers for the tests I was running I checked them. It was all fine 4. Threading. I disabled threading to see if that was the issue. It wasn't 5. Performance monitor: what was happening? It turns out that tens of thousands of objects were being created where before merely hundreds were. What the...? There was definitely extra processing going on. But where and why? 6. Step through everything. I was doing the previous tests on our staging server to simulate the live scenario so now I switched to the my home machine and had a look. It was fast. Lightning fast. Object creation was normal and everything perfect. Ooohkaaay.... 7. Rebuild and Redeploy. Ensuring I got the latest code from the source code repository I did a clean and full rebuild on my office machine (better connection to the hosting centre from there) and deployed. Retest and nothing. OK, this is weird. 8. Get desperate The version on my local machine at home works fast. The live version works slow. Try uploading the version from my home machine and not my office machine. It works! But they are from the same code base...? Now I'm scared. And then I saw it. A small file called 'ErrorLog.txt' that works as our last line of defence in error trapping was being created by the live version. The file contained thousands of messages about a missing config value, but the error was so deep it didn't even get the chance to tell me which value it was. Bring out WinMerge and check all config files and there we have it. A value in the config files was different Local version: SubtleBugsId Live version: SubtleBugId It turns out the live version used t
Chris Maunder wrote:
Bring out WinMerge and check all config files and there we have it. A value in the config files was different
Don't feel bad. The only issue today cropped up when commands were sent to rotate the robot's waist joint. The joint moved in the opposite direction from what flight controllers expected. Alibaruho said engineers suspect something simple, possibly an incorrect sign in a configuration file, may be to blame. -- link[^] You are now qualified to write robotics software for the ISS. :-D Marc
-
Ri Qen-Sin wrote:
all the e-mails
Depends which list. We were running at around 600,000 emails an hour on the daily Insider today. A new land-speed record for us but we've not really optimised our setup that much and there's still more scope for speed improvements. Our previous mailout system pushed 13,000 an hour. I love C#.
cheers, Chris Maunder
CodeProject.com : C++ MVP
Don't let the botnet controllers find out, or you'll be pushing 1M and not all of them about CodeProject.
Ninja (the Nerd)
Confused? You will be... -
Please read the forum guidelines. This isn't an example of a subtle bug. It's an example of a mistake, an error or oversight. Sofware wasn't to blame....the programmer was! Just joking :laugh: Been down that path too. Venting frustration outside, yelling WTF and occasionally throwing objects against other harder objects. Then epiphany and :doh: :-\ Nice job tracking it down.
src_Objects being members of your anatomy and dest_Objects being of the building? void ApplyMeaninglessForceUntoWall() { return pain; }
Ninja (the Nerd)
Confused? You will be... -
Chris Maunder wrote:
Bring out WinMerge and check all config files and there we have it. A value in the config files was different
Don't feel bad. The only issue today cropped up when commands were sent to rotate the robot's waist joint. The joint moved in the opposite direction from what flight controllers expected. Alibaruho said engineers suspect something simple, possibly an incorrect sign in a configuration file, may be to blame. -- link[^] You are now qualified to write robotics software for the ISS. :-D Marc
Coming soon to an airport near you... Plane Digs Into Ground Instead Of Taking Off Badly signed config file blamed for $X million of damage done.
Ninja (the Nerd)
Confused? You will be... -
This is called Irony. Our mailout system suddenly decided to go slow. Normally we can send the newsletter out at 200K - 300K emails an hour but today it was on a go-slow and was pushing 3,000 an hour. With nearly a million subscribers waiting to hear the news this wasn't looking good yet I had no idea what was going wrong. First the obvious things: 1. Check the database. Everything seemed OK, in that it was slow across our live and all our test servers. Some of these servers are still in the pre-2.2 code state so if it was a 2.2 code issue it would hopefully show up there. Nothing. 2. Check the code. Again, manual checks of all relevant code to see what the changes between 2.1 and 2.2 code were and nothing, apart from the known and tested changes, was showing up. 3. Check the mail servers. Even though I wasn't using any mail servers for the tests I was running I checked them. It was all fine 4. Threading. I disabled threading to see if that was the issue. It wasn't 5. Performance monitor: what was happening? It turns out that tens of thousands of objects were being created where before merely hundreds were. What the...? There was definitely extra processing going on. But where and why? 6. Step through everything. I was doing the previous tests on our staging server to simulate the live scenario so now I switched to the my home machine and had a look. It was fast. Lightning fast. Object creation was normal and everything perfect. Ooohkaaay.... 7. Rebuild and Redeploy. Ensuring I got the latest code from the source code repository I did a clean and full rebuild on my office machine (better connection to the hosting centre from there) and deployed. Retest and nothing. OK, this is weird. 8. Get desperate The version on my local machine at home works fast. The live version works slow. Try uploading the version from my home machine and not my office machine. It works! But they are from the same code base...? Now I'm scared. And then I saw it. A small file called 'ErrorLog.txt' that works as our last line of defence in error trapping was being created by the live version. The file contained thousands of messages about a missing config value, but the error was so deep it didn't even get the chance to tell me which value it was. Bring out WinMerge and check all config files and there we have it. A value in the config files was different Local version: SubtleBugsId Live version: SubtleBugId It turns out the live version used t
Actually, that raises an issue with .NET that bugs me. Opening a text file for append is dog slow when the file is of any size, no matter what method you use. I think the .NET framework must use a very inefficient (character-at-a-time, probably) method to read through to the end of the file (when it could, of course, just do a simple seek). I suppose this was lucky for you though in a way in that the performance impact this causes made you go looking for your bug.
Paul Sanders http://www.alpinesoft.co.uk