Worst deploy story?
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
Chris Maunder wrote:
I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad.
I have cleaned up after plenty of people who have had deployments go sideways. So far, knock on wood I have always had a good back out plan.
Common sense is admitting there is cause and effect and that you can exert some control over what you understand.
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
Back in the day, the company I worked for wrote software that was "deployed" by a custom-cut CD to around 100 remote locations around the state. We had no control over their environment (other than to specify it was PC/windows 9x). The bloke developing that software cut the cd's for go live, then went on a month's vacation - overseas and unreachable. Needless to say, the go live didn't go very well, and yours truly was tasked with cleaning up the mess - some very, very long hours over a few days to get it under control enough to cut a new set of cd's and try again, followed by the remainder of the month working on performance issues. (OT: under a quirk of the wage agreement I was working under, because I didn't have enough of a break between going home and returning over a few days, by the third day I was working on QUADRUPLE time... go me!!) The bloke returned from holidays, dropped a resignation letter on the manager's desk and walked out again.
Quad skating his way through the world since the early 80's... Booger Mobile - My bright green 1964 Ford Falcon - check out the blog here!! | If you feel generous - make a donation to Camp Quality!!
-
Back in the day, the company I worked for wrote software that was "deployed" by a custom-cut CD to around 100 remote locations around the state. We had no control over their environment (other than to specify it was PC/windows 9x). The bloke developing that software cut the cd's for go live, then went on a month's vacation - overseas and unreachable. Needless to say, the go live didn't go very well, and yours truly was tasked with cleaning up the mess - some very, very long hours over a few days to get it under control enough to cut a new set of cd's and try again, followed by the remainder of the month working on performance issues. (OT: under a quirk of the wage agreement I was working under, because I didn't have enough of a break between going home and returning over a few days, by the third day I was working on QUADRUPLE time... go me!!) The bloke returned from holidays, dropped a resignation letter on the manager's desk and walked out again.
Quad skating his way through the world since the early 80's... Booger Mobile - My bright green 1964 Ford Falcon - check out the blog here!! | If you feel generous - make a donation to Camp Quality!!
_Damian S_ wrote:
The bloke returned from holidays, dropped a resignation letter on the manager's desk and walked out again.
Oh man. That's cold.
cheers Chris Maunder
-
_Damian S_ wrote:
The bloke returned from holidays, dropped a resignation letter on the manager's desk and walked out again.
Oh man. That's cold.
cheers Chris Maunder
No great loss, he was a douchebag anyway!! Funny story, after he left, we had to clear out his desk, and in it, found a hand written list of porn titles, around 1/2 of which were crossed off... At least that explained the lunches away from his desk and returning smelling a little funky... :laugh: :laugh:
Quad skating his way through the world since the early 80's... Booger Mobile - My bright green 1964 Ford Falcon - check out the blog here!! | If you feel generous - make a donation to Camp Quality!!
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
I once worked on a support site for a computer manufacturer that saved over $300,000 a day in phone support costs. I did a deploy that caused tons of lockups, server reboots and downtime for a few days afterward. We were pulling our hair out trying to figure out the problem, and had no idea. We eventually talked to someone in tech support, and they let us know that one of the popular pages would hang if you clicked go without entering a serial number. They let me know they had a web page spinning on it to see if it would return all the systems. :sigh: I had removed the front end validation because I thought it was redundant to check what the COBOL service running on the mainframe was going to check anyway. It wouldn't have been a problem, but there were multiple entries in the table with missing serial numbers and the service expected the index to be unique. Needless to say multiple patches to multiple layers of software were quickly deployed.
Curvature of the Mind now with 3D
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
Two such events come to mind. One was an OS upgrade -- VOS (my candidate for The World's Worst Operating System) on a Stratus System 2000, circa 1994, this was (maybe still is) a system that processed student loan bills for like the whole country. We were upgrading (in order to support Ethernet! Whoo hoo!) and over a few months prior, many Consultants had checked out the existing system to ensure that everything would go smoothly, nothing would go wrong. I was tasked with performing the upgrade. The Boss had said to call him every two hours, or immediately if there was a problem. On that fated Friday evening I entered the server room, performed a full backup, and began the upgrade. At the end of the upgrade a reboot (of course). Then nothing. Nothing? Just a blinking cursor on the terminus (VT-100 clone). Turn-it-off-turn-it-back-on, jiggle-the-connector. Nothing. Check the baud, bits, parity, etc. Nothing. Call the Boss. Couldn't even restore the backup without console access. Nothing. Call our friendly neighborhood Stratus tech who was also on-call. Tech arrives. Nothing. Calls others. Burns incense, etc. I don't know how they got it going again, but eventually, more than a day later, it was working just fine. As I understand it, the previous version of VOS allowed the console to connect with 9600,8,N,1 (same as my cherished DEC systems), but the new version required that the console settings be set specifically in the startup script (WTF?!) -- our startup script didn't contain such a statement. Had this been detected by the consultants during the months of validation, one could have been added. Without it, the system simply refused to communicate with the console. I don't know how they got it working, but I'll always remember the night I spent twenty-eight hours in the server room. Two was an OS licensing boondoggle -- QNX on custom hardware (let's call them Point Of Sale Systems for simplicity), late 90s. We provided a bunch of hardware to a client. The hard drives had all been cloned, so they all had the same QNX license. We had a Big Box of Licenses (each on a 3.5" floppy) that had been purchased for the hardware, but expediency meant we shipped the systems as they were (in a working state) and we'd see about applying the licenses at some time later. Later, I was tasked with providing a script to the tech who would take the Big Box of Licenses and apply them. I wrote the script, I tested the script, it worked just fine. Several times. In the lab. I shipped the Big Box of Li
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
We're about to deploy a major major upgrade to the new system. I'm IT manager, and paranoid, so everything is being tested, tested, then tested again. This involves using various test pre-deployment databases to run the various update scripts against, then doing a system test against the updated database. There were a lot of database schema changes. A lot. Somewhat more of a conversion than an update in some cases. And the Address table was particularly problematic -as we were introducing encoding using the Post Office's Unique Postal ID system - so as it converted the address to the new schema, it used the P.O. software to try to find the matching address. The ones thrown out were being manually cleaned by a team of cannon fodder - then the conversion run again - we'd agreed not to do a final convert until we could get 95% hit rate. A contractor was assigned the job of tweaking and testing the conversion. Everything else was ready, and we still had 2 days before the rollout. So there was a relaxed atmosphere. I'm sitting, chatting to one of the devs, and I notice the contractor standing at the end of my desk. I finish the chat, turn to the contractor, who says "Maxxx. I just truncated the Address table" "That's OK" quoth I, "it's just the pre-deployment database, we can recreate it easy enou.." That's when I noticed his head was shaking, slowly, from side to side. "Production" he croaked. That's when my phone rang. 150 phone operators had suddenly lost every address in the system. During the next 48 hours the contractor, myself, and one of the contractors friends, managed (just) to restore the address table from a backup. Let me tell you, that was not in the least bit easy! And scary! Using undocumented features of Oracle. After I had bought beers I suggested to the contractor that he might want to make the background colour of his Live vs Pre-Deployment windows a different shade - maybe FLASHING RED for the live database! It's funny, but although at the time it was very stressful, looking back it was actually bloody good fun! time is a great healer!
PooperPig - Coming Soon
-
We're about to deploy a major major upgrade to the new system. I'm IT manager, and paranoid, so everything is being tested, tested, then tested again. This involves using various test pre-deployment databases to run the various update scripts against, then doing a system test against the updated database. There were a lot of database schema changes. A lot. Somewhat more of a conversion than an update in some cases. And the Address table was particularly problematic -as we were introducing encoding using the Post Office's Unique Postal ID system - so as it converted the address to the new schema, it used the P.O. software to try to find the matching address. The ones thrown out were being manually cleaned by a team of cannon fodder - then the conversion run again - we'd agreed not to do a final convert until we could get 95% hit rate. A contractor was assigned the job of tweaking and testing the conversion. Everything else was ready, and we still had 2 days before the rollout. So there was a relaxed atmosphere. I'm sitting, chatting to one of the devs, and I notice the contractor standing at the end of my desk. I finish the chat, turn to the contractor, who says "Maxxx. I just truncated the Address table" "That's OK" quoth I, "it's just the pre-deployment database, we can recreate it easy enou.." That's when I noticed his head was shaking, slowly, from side to side. "Production" he croaked. That's when my phone rang. 150 phone operators had suddenly lost every address in the system. During the next 48 hours the contractor, myself, and one of the contractors friends, managed (just) to restore the address table from a backup. Let me tell you, that was not in the least bit easy! And scary! Using undocumented features of Oracle. After I had bought beers I suggested to the contractor that he might want to make the background colour of his Live vs Pre-Deployment windows a different shade - maybe FLASHING RED for the live database! It's funny, but although at the time it was very stressful, looking back it was actually bloody good fun! time is a great healer!
PooperPig - Coming Soon
When I read the first couple of lines, I thought this story was current... ;-)
Quad skating his way through the world since the early 80's... Booger Mobile - My bright green 1964 Ford Falcon - check out the blog here!! | If you feel generous - make a donation to Camp Quality!!
-
We're about to deploy a major major upgrade to the new system. I'm IT manager, and paranoid, so everything is being tested, tested, then tested again. This involves using various test pre-deployment databases to run the various update scripts against, then doing a system test against the updated database. There were a lot of database schema changes. A lot. Somewhat more of a conversion than an update in some cases. And the Address table was particularly problematic -as we were introducing encoding using the Post Office's Unique Postal ID system - so as it converted the address to the new schema, it used the P.O. software to try to find the matching address. The ones thrown out were being manually cleaned by a team of cannon fodder - then the conversion run again - we'd agreed not to do a final convert until we could get 95% hit rate. A contractor was assigned the job of tweaking and testing the conversion. Everything else was ready, and we still had 2 days before the rollout. So there was a relaxed atmosphere. I'm sitting, chatting to one of the devs, and I notice the contractor standing at the end of my desk. I finish the chat, turn to the contractor, who says "Maxxx. I just truncated the Address table" "That's OK" quoth I, "it's just the pre-deployment database, we can recreate it easy enou.." That's when I noticed his head was shaking, slowly, from side to side. "Production" he croaked. That's when my phone rang. 150 phone operators had suddenly lost every address in the system. During the next 48 hours the contractor, myself, and one of the contractors friends, managed (just) to restore the address table from a backup. Let me tell you, that was not in the least bit easy! And scary! Using undocumented features of Oracle. After I had bought beers I suggested to the contractor that he might want to make the background colour of his Live vs Pre-Deployment windows a different shade - maybe FLASHING RED for the live database! It's funny, but although at the time it was very stressful, looking back it was actually bloody good fun! time is a great healer!
PooperPig - Coming Soon
_Maxxx_ wrote:
he might want to make the background colour of his Live vs Pre-Deployment windows a different shade
Reminds me of a time when a Dell sales manager came and bitched at me that my system was not storing his data, we implemented a 2cm bright orange border on all UAT forms after that. And I still do 15 years later!
Never underestimate the power of human stupidity RAH
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
It doesn't happened to me, but a mate from the office... Those days we had no direct connection to customers so deployment involved going to the site physically... We had a set of 18 1.44 floppy disks we were preparing for deployment. My mate prepared the disks, took them and went home. At the morning he went on for a 3 hour driving to get to the customer, just to find, that the last floppy still in the PC at the office... After some thinking over phone we send him to buy a modem (28.8 was the best back then), install it on one of the customers PCs and set up a connection to transfer the file... After an other hour and half he had all the material at hand and with a 3 hours delay started the deployment... And we got our ever first online customer!!!
Skipper: We'll fix it. Alex: Fix it? How you gonna fix this? Skipper: Grit, spit and a whole lotta duct tape.
-
Back in the day, the company I worked for wrote software that was "deployed" by a custom-cut CD to around 100 remote locations around the state. We had no control over their environment (other than to specify it was PC/windows 9x). The bloke developing that software cut the cd's for go live, then went on a month's vacation - overseas and unreachable. Needless to say, the go live didn't go very well, and yours truly was tasked with cleaning up the mess - some very, very long hours over a few days to get it under control enough to cut a new set of cd's and try again, followed by the remainder of the month working on performance issues. (OT: under a quirk of the wage agreement I was working under, because I didn't have enough of a break between going home and returning over a few days, by the third day I was working on QUADRUPLE time... go me!!) The bloke returned from holidays, dropped a resignation letter on the manager's desk and walked out again.
Quad skating his way through the world since the early 80's... Booger Mobile - My bright green 1964 Ford Falcon - check out the blog here!! | If you feel generous - make a donation to Camp Quality!!
I enjoyed reading this as much as I pitied you. Try script writing my friend but remember the end should be a better one than the story is.
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
What is until today still known as the "Ghent Crisis". A bug that creeped up after roll-out and eluded us on trying to reproduce it. There was a severe political deadline (and thus media attention upon failure). Me and my entire team of about 5 people had to go on site from 08:00-23:00 every day (including weekends) until the bug was fixed. My boss finally managed to reproduce the bug, but it still took us several days to fix it due to it's complexity. At least we all got a small bonus for it. :-D ([EDIT]The entire crisis lasted for two weeks and we managed in not having a roll back.[/EDIT])
V.
(MQOTD rules and previous solutions) -
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
When you set up a retail banking system, it is imperative that you test ALL the posting possibilities of the General Ledger BEFORE deploying. I was working at a rather large bank and we were going live with the new retail system. Friday night, unbeknown to us, the back office team in charge of config and data make an itsie teeny weeny change to one of the GL posting rules. Data migration flew past over the weekend as it had been properly tested and we were good to go for Monday morning's go live. Then. Things. Happened. Luckily we had a couple of guys in branch to watch what was happening and we got a call that the first transactions where going through. So to be sure, we checked everything was going okay. Cash books - check, account balances - check, client [GL] accounts - check, bank's ledgers - FAARRRKKKK!!!!! what should be impossible - mismatched entries - where all over the GL. Feck, feck, feckity. We couldn't shut down to investigate, so a couple of poor sods from back office start manually entering corrections. Then a bright spark asks "Where have the other postings gone? They have to be 'somewhere'?" All the missing postings where found in a back drawer correction ledger and we had to manually back out all the earlier corrections and correct the original incorrect postings all while not knowing where the hell the mismatches where being generated from. When the bad rule was found the guy who changed it was castigated some what. This then became a near standard feature of any roll out for them, at the eleventh hour someone would add in a little spice and it all go boom. :sigh:
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
Many years ago I worked for a company that had over a hundred Linux servers around the country running Progress DB and application. Each of these servers where connected by a dial up modem that we could also dial into (providing they were still plugged into the same phone line that we had installed them on). We needed to make a schema change to each of the DBs which were all identical. We had no way of making such changes, although we did for code and data. We came up with a script that could be deployed, pulled down by each server when it made a routine connection, and then run to make the schema change. We tested it, it worked. I released it at 6 o'clock on a Saturday morning when I was the only person in the office and sat back to watch the monitor software for signs of the change being successfully applied. One by one the updates failed, then people started to phone up because they could no longer use their system. I phoned up the big boss man, we had a quick chat, we worked out what had gone wrong and how to fix it. The fix involved me dialing into each server, running a few commands against the DB, then starting it back up again. In just over a hundred servers three applied the patch correctly. By the time I had done fixing the rest I went to go home, locked up the warehouse I was in, set the alarm (I was the only person working in it on that Saturday), then went to leave the site. When I got to the gate I found them padlocked and the the site empty. I had to phone the big boss man back who made a few calls and found someone to come down with some bolt cutters to set me free. There then followed a comprehensive review of the company's lone working procedures (of which there had been none before).
Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them.
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
While it's not specifically related to software deployment, the first thought that came to my mind is that anyone in the business of launching satellites and people into space have truly the worst deployment stories. Marc
Imperative to Functional Programming Succinctly Higher Order Programming
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
In the last '80s, I was working for a company that was running two production systems on PDP 11/73's in a manufacturing environment. We had maxed out the memory and had code that would work in test, but not in production because of the memory issues. The first system was product routing in the plant; the second was a storage system for receipt/delivery from/to the first system. A project was created to rewrite the first system to run on a MicroVAX; similar technology, language, etc. The rewrite took a year with a team of people. When implemented (and there was no go back, only go forward), it was discovered the code had not been complied with array bound check. It wasn't done in the old version because that would use to much memory and overlooked in the new system. Production ground to a halt for January and most of February. Changes had to be described on a paper form and signed off by on site support before being implemented. The next year, the second system was rewritten with lessons learned. On implementation, around the clock support. Management asked when it was going to be installed; we said it was installed - two weeks ago. Much better. I am a firm proponent to a post mortem on all projects; see what worked, what didn't and learn from it. No finger pointing, just learning. Tim
-
While it's not specifically related to software deployment, the first thought that came to my mind is that anyone in the business of launching satellites and people into space have truly the worst deployment stories. Marc
Imperative to Functional Programming Succinctly Higher Order Programming
Yeah. They win this game. :sigh:
cheers Chris Maunder
-
OK, so we had a messy, messy day here with a deploy that went pear-shaped, so I was wondering if anyone has any true horror stories of deploys that went terribly, horrifyingly bad. The sorts of thing where you no longer even visit that town because the Wanted posters are still flying from the street posts.
cheers Chris Maunder
Great Thread. I love ghost stories.
-
Great Thread. I love ghost stories.
