An eye opener for all you workaholics out there...

charlieg

:) you plagiarized that to an extent. Decades ago I worked in a shop that had a rule - you shut down the wrong server, you buy lunch for the team. There I am rebooting my workstation, except I was logged in to the main server with 2k+ users on it.... yeah, I bought lunch.

Charlie Gilley “They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759 Has never been more appropriate.

Sander Rossel

I plagiarized that to an extent of 100% from someone who probably did the same :laugh:

charlieg wrote:

There I am rebooting my workstation

Classic! :laugh: We had a client once, small to medium sized meat processor, who wasn't very keen on IT... Their server, which kept the whole factory going, was a regular PC standing under someone's desk. I think the PC was not a regular workstation, but that wasn't necessary for what happened. The guy at the desk simply bumped into it with his knee and the whole factory fell silent. They called us in, we went to the site, only to find the server shut down. Guy hit the off button :laugh:

Best, Sander Azure DevOps Succinctly (free eBook) Azure Serverless Succinctly (free eBook) Migrating Apps to the Cloud with Azure arrgh.js - Bringing LINQ to JavaScript

honey the codewitch

A friend of mine was dogged by slow Internet, well below the service levels he paid for. Since it was DSL based he figured it was phone company nonsense, and was an ongoing battle with the phone company over his service. He couldn't or wouldn't switch to cable, although I don't remember why. Anyway, eventually it comes up in conversation that he has a Windows 2008 Server machine that is Internet facing, running out of his home network. *headdesk* As soon as he said that, I was like "here's how to fix your Internet" Sure enough, half the network was basically a botnet for Kevin only knows who. He doesn't run servers out of his home anymore.

To err is human. Fortune favors the monsters.

obermd

I entered kill 0 (instead of kill 9) on a SunOS server one time and it failed to come back up. The problem is it was a core server for a large number of administrative people. Turns out it had previously corrupted the file system. It was a week before it was available again.

kmoorevs

This is a mostly thankless profession when things work and a barrage of complaining/whining when they don't! :laugh: I currently have too many people needing too many things all at once, reminding me of a recurring nightmare...a queue of angry customers, colleagues, and relatives lined up at my desk, waiting for me to 'fix their problems'. :omg: Current project: Ony 131 errors left and already a 90-minute call about it and what is taking so long! :confused:

"Go forth into the source" - Neal Morse "Hope is contagious"

PIEBALDconsult

I accidently tested a database shutdown script I was developing on the PROD system, but who hasn't done that? One day I found out that a DEC AlphaServer 1000 has a switch which kills power if the case is opened while the system is running. I mean, WTF? Definitely over-engineered. All I was doing was determining whether or not there were any spare RAM slots. : sheesh : At least that was a DEV system.

Mycroft Holmes

Also decades ago I worked for a rather large broking company. Sysadmin could not work out who owned a particular server (there were a lot of servers) so he shut it down to see who screamed. It turned out to be a company that had been sold off 4 years before and moved 2 floors down.

Never underestimate the power of human stupidity - RAH I'm old. I know stuff - JSOP

FreedMalloc

Ah, the 'ol scream test. An oldie but a goodie. But, potentially career limiting if the wrong person screams.

Nelek

My own experience was already commented so I won't repost. But... we (and another 4 factories) had around 6 weeks of isolation from the mother factory (no changes in the library, no backups of Prod-DB, no...) just because the main IT Department changed the IP of the production servers, without notifying it anywhere (not even locally) :mad::mad::mad: we were the first ones asking "why doesn't work anymore?" that triggered the look up of the colleagues at the main company. I had to submit around 280 change requests to each one of the 3 different firewalls in the factory nodes on the way to us... :doh: :doh: :doh: X| X| X|

M.D.V. ;) If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about? Help me to understand what I'm saying, and I'll explain it better to you Rating helpful answers is nice, but saying thanks can be even nicer.

GuyThiebaut

One chap I was called to help around 25 years ago explained to me that every time he pressed a key on his keyboard his computer would restart. So I went over to his desk and observed what was happening and yes the computer did shut down every time he pressed a key on the keyboard. Now my Sherlock Holmes senses kicked in as I noticed that he also had a folder resting on his lap. Every time he leaned forward to press a key, on the keyboard, the corner of the folder pressed the restart button on his computer...

“That which can be asserted without evidence, can be dismissed without evidence.”

― Christopher Hitchens

Mike Winiberg

Hmm, Done both of those! 8) Worst for me was - on a production system - deleting an 'old' copy of a database from the backup filestore during a very busy morning. You guessed it - not the 'old' copy at all! This was all long before SQL (or even Linux) was a thing on PCs. Had to restore from the overnight tape backup (we did have backups, and they were regularly tested, fortunately) and then get the reservation staff to go through all the paper printouts of that day's bookings to recover the mornings transactions, whilst suspending new ones to guard against double bookings. (Fortunately I also produced a paper copy of every transaction as - back in the day - neither the power nor tape backups were 100% reliable). I was not Mr popular that day, I can tell you! I learnt my lesson though and added an hourly local backup to a .zip file of transaction data just in case. That saved my bacon a few times too!

CodeZombie62

Back in the mid-80’s at a former employer who did photo processing they did a major upgrade of the DEC PDO-11/70 systems located in their multiple film processing plants to Microvax. Each of the new Microvax systems was setup with two large hard drives with one being mirrored on the other. They were hot swappable so if something went wrong with one drive they could stop it and keep going with the other one while the bad drive was sent for repairs. So one morning at work we were alerted to a problem that a bunch of invoices were missing in one of the plants. It turned out that at this particular plant they had to get a hard drive repaired and when it came back a few days later they started mirroring the drives again only someone (not me, thankfully) did the mirroring in the wrong direction so the drive that had the old data overwrote the drive with the current data. I’m not sure what they ultimately did to get the data back since I don’t think there was a tape backup because of the disk mirroring (I think it was probably the same person who thought they no longer needed the redundant DECNet cross links between the plants) but I’m thinking one of the devs had to write something so some poor workers in the plant could do a bunch of data entry.

Cpichols

Congrats on your immortality! And welcome to the club :laugh:

Keefer S

I've done it more than once. The screams usually come pretty quickly.

Rich Shealer

About 25 years ago I learned an axiom that I live by. "It only takes one "Oh Sh*t!" to wipe out 25 "Atta boy"s.

Kirk 10389821

I once deleted ALL of the car loans for FL because someone changed the "mechanical" terminal Switch, and I was NO LONGER on the DEV box. OUCH! A month later, someone else did a different state. And I patched the operating systems for production to block the command (renamed it to another command we would NEVER type accidentally)...

Paul Sanders the other one

Good post. There is *absolutely no point* working when you're tired. You get next to nothing useful done and you make mistakes. It also leads to burnout, which can be devastating. It (burnout) happened to me and also to a friend of mine, and it put us both in hospital (at different times). It took me a long time to figure all this out, but I've got it now. I work for myself nowadays so I can do what I like, but more employers need to understand this and to take proper care of the mental health of their staff. Sorry Sander, maybe a bit more than you were expecting, but I feel very strongly about this. PS: Take breaks. Take breaks, take breaks, take breaks.

Paul Sanders. If I had more time, I would have written a shorter letter - Blaise Pascal. Some of my best work is in the undo buffer.

obermd

I've had to do this as well since no one seems to know what their resources are.

James G 2021

My first job was 100% Assembler. In an effort to squeeze out a few more bytes of memory for variables, I was cleaning up a program and deleted a single space located before the variable used in the print command. The program worked fine but, printed a few boxes of pre-printed forms with garbage due to the preceding valid blank space being removed leaving another character to be mvcl'd across the print line before it was built. Almost a career limiting move.

Paul Sanders the other one

Hard work, assembler. I'm too old now to have the focus you need (but there are compensations). My favourite oh no! story is from my college days, programming on punch cards (ouch, that really dates me). Somehow, a rogue card sneaked its way into the deck and as a result my printout consisted of 2000 or so blank pages. I bet the ops loved that!

Paul Sanders. If I had more time, I would have written a shorter letter - Blaise Pascal. Some of my best work is in the undo buffer.