Scary Responsibility?
-
I work in automation systems so it's a fairly common-place thing for me to have a lot of scary responsibility of many varieties. Probably the most scary had to do a financial responsibility. I was working on a semiconductor processing system that was dealing with the chemical processing of wafers. These systems were very, very time-sensitive. We had several cassettes, each full of fifty wafers in process at a time in the system. One type of system dealt with acids that would etch off oxide that had been deposited on the wafers. Most customers would dilute the acid so the etching took longer but it was rather tolerant to immersion time variances. Other customers insisted on very high acid concentration which etched very fast and were very sensitive to the immersion time. It had to be accurate to the second. If the time was off by more than a second then the wafers over or under etched and the entire cassette was scrapped which resulted in the loss of millions of dollars of product. We spent an immense amount of time making sure that the entire process was exactly accurate. Imagine my frustration at how poorly the customer trained their operators. I had already been instructed to assume a sixth grade reading level from the operators and at times that seemed optimistic. I remember vividly watching an operator and guiding them during the first shift of production. We had to use a flaky wafer transfer unit that would load the cassettes into the system, as required by the customer, and these things would give errors often. When that happened the operator was supposed to cancel the transfer in progress and try again. Instead, I watched the operator press abort three times in the first shift, causing approximately ten million dollars in scrapped product. I was then instructed to put a double level of "are you sure" confirmations instead of just the one. I then had to tell them not to even use abort except for very special circumstances like power loss when the UPS couldn't keep the robots alive because the batteries were too weak. That was very, very frustrating. One amusing thing happened when we were installing a system in Erfurt, Thuringia, Germany. We were undergoing our final confirmation testing and the process engineer told us the etch times needed to be accurate to within one second. He emphasized this by saying, "and I mean German seconds, not American seconds." I got a good chuckle from that. When they approved the system he gave us a stopwatch that had been made in the USSR since Thur
How does a German Second compare to a New York Minute? *raises eyebrow*
Real programmers use butterflies
-
I work in automation systems so it's a fairly common-place thing for me to have a lot of scary responsibility of many varieties. Probably the most scary had to do a financial responsibility. I was working on a semiconductor processing system that was dealing with the chemical processing of wafers. These systems were very, very time-sensitive. We had several cassettes, each full of fifty wafers in process at a time in the system. One type of system dealt with acids that would etch off oxide that had been deposited on the wafers. Most customers would dilute the acid so the etching took longer but it was rather tolerant to immersion time variances. Other customers insisted on very high acid concentration which etched very fast and were very sensitive to the immersion time. It had to be accurate to the second. If the time was off by more than a second then the wafers over or under etched and the entire cassette was scrapped which resulted in the loss of millions of dollars of product. We spent an immense amount of time making sure that the entire process was exactly accurate. Imagine my frustration at how poorly the customer trained their operators. I had already been instructed to assume a sixth grade reading level from the operators and at times that seemed optimistic. I remember vividly watching an operator and guiding them during the first shift of production. We had to use a flaky wafer transfer unit that would load the cassettes into the system, as required by the customer, and these things would give errors often. When that happened the operator was supposed to cancel the transfer in progress and try again. Instead, I watched the operator press abort three times in the first shift, causing approximately ten million dollars in scrapped product. I was then instructed to put a double level of "are you sure" confirmations instead of just the one. I then had to tell them not to even use abort except for very special circumstances like power loss when the UPS couldn't keep the robots alive because the batteries were too weak. That was very, very frustrating. One amusing thing happened when we were installing a system in Erfurt, Thuringia, Germany. We were undergoing our final confirmation testing and the process engineer told us the etch times needed to be accurate to within one second. He emphasized this by saying, "and I mean German seconds, not American seconds." I got a good chuckle from that. When they approved the system he gave us a stopwatch that had been made in the USSR since Thur
Rick York wrote:
magine my frustration at how poorly the customer trained their operators. I had already been instructed to assume a sixth grade reading level
I, I kid you not, had to change all text in a version of the software to add recognizable pictograms to any prompt or error because a large part of the working force, QA in a pharmaceutical company, was totally illiterate.
GCS d--(d-) s-/++ a C++++ U+++ P- L+@ E-- W++ N+ o+ K- w+++ O? M-- V? PS+ PE- Y+ PGP t+ 5? X R+++ tv-- b+(+++) DI+++ D++ G e++ h--- r+++ y+++* Weapons extension: ma- k++ F+2 X
-
Back when I worked for an aircraft company I was responsible for coding a set of traffic lights in the telemetry/testing office they were mounted upside down with green at the top. If certain things went over certain limits I was to make the amber light come on, further exceedance would cause the red light to come on at which point the aircraft controller would yell, "Eject, eject, eject", over the radio and the pilot would use the ejection seat to bail out before the multi-million dollar plane exploded. Lots of testing ensured the thing would work (feeding fake data from a recorded wire). I felt this was an enormous responsibility for a young (at the time) programmer to assume, especially since they had no peer reviewing mechanism or anything other than testing done by oneself. A life and lots of expensive equipment, not to mention where the bits crashed/fell were down to me getting it right. It was used once (correctly) while I was on vacation and twice more after I left the company; so all good. In a later job I programmed a software/hardware automation system. We nearly took a contract monitoring a nuclear power plant but the president of the company objected to the cost of the insurance policy we had to take as part of the deal and backed out. I was pleased about this although I had faith in our system I was happy to NOT have that responsibility. Has anyone else been in a similar situation?
- I would love to change the world, but they won’t give me the source code.
Not on the same scale at all, but at one time (1980s, when I was in my mid-20s) I worked on a credit-card processing system. It was all batch COBOL stuff and ran for many hours each night. Myself and a colleague took turns on overnight support, which involved using a portable dial-up teletype to get a core dump, from which we diagnosed any problem and either corrected faulty input or coded up a workaround. One day my colleague dealt with a problem but inadvertently used a comma, not a full stop, in one correction. I don't know the exact problem, but suffice to say that during the review next day we realised he'd "lost" a little over £12,000. He was mortified and for the next few weeks I was petrified when making code changes at 3am on my own... (Sitting on the floor in my hallway because the modem lead wasn't long enough).
-
Back when I worked for an aircraft company I was responsible for coding a set of traffic lights in the telemetry/testing office they were mounted upside down with green at the top. If certain things went over certain limits I was to make the amber light come on, further exceedance would cause the red light to come on at which point the aircraft controller would yell, "Eject, eject, eject", over the radio and the pilot would use the ejection seat to bail out before the multi-million dollar plane exploded. Lots of testing ensured the thing would work (feeding fake data from a recorded wire). I felt this was an enormous responsibility for a young (at the time) programmer to assume, especially since they had no peer reviewing mechanism or anything other than testing done by oneself. A life and lots of expensive equipment, not to mention where the bits crashed/fell were down to me getting it right. It was used once (correctly) while I was on vacation and twice more after I left the company; so all good. In a later job I programmed a software/hardware automation system. We nearly took a contract monitoring a nuclear power plant but the president of the company objected to the cost of the insurance policy we had to take as part of the deal and backed out. I was pleased about this although I had faith in our system I was happy to NOT have that responsibility. Has anyone else been in a similar situation?
- I would love to change the world, but they won’t give me the source code.
For this reason alone (just kidding, I love games) I went to videogame programming. No interaction with clients and nobody dies. But coming of age I've become more stoic about it. Civil and aeronautic engineers have a hell of a lot more lives on their shoulders...and they seem to shrug off the what ifs pretty well.
-
Rick York wrote:
magine my frustration at how poorly the customer trained their operators. I had already been instructed to assume a sixth grade reading level
I, I kid you not, had to change all text in a version of the software to add recognizable pictograms to any prompt or error because a large part of the working force, QA in a pharmaceutical company, was totally illiterate.
GCS d--(d-) s-/++ a C++++ U+++ P- L+@ E-- W++ N+ o+ K- w+++ O? M-- V? PS+ PE- Y+ PGP t+ 5? X R+++ tv-- b+(+++) DI+++ D++ G e++ h--- r+++ y+++* Weapons extension: ma- k++ F+2 X
-
Back when I worked for an aircraft company I was responsible for coding a set of traffic lights in the telemetry/testing office they were mounted upside down with green at the top. If certain things went over certain limits I was to make the amber light come on, further exceedance would cause the red light to come on at which point the aircraft controller would yell, "Eject, eject, eject", over the radio and the pilot would use the ejection seat to bail out before the multi-million dollar plane exploded. Lots of testing ensured the thing would work (feeding fake data from a recorded wire). I felt this was an enormous responsibility for a young (at the time) programmer to assume, especially since they had no peer reviewing mechanism or anything other than testing done by oneself. A life and lots of expensive equipment, not to mention where the bits crashed/fell were down to me getting it right. It was used once (correctly) while I was on vacation and twice more after I left the company; so all good. In a later job I programmed a software/hardware automation system. We nearly took a contract monitoring a nuclear power plant but the president of the company objected to the cost of the insurance policy we had to take as part of the deal and backed out. I was pleased about this although I had faith in our system I was happy to NOT have that responsibility. Has anyone else been in a similar situation?
- I would love to change the world, but they won’t give me the source code.
� Forogar � wrote:
Has anyone else been in a similar situation?
My first bug in the wild wasn't scary, until things went wrong. Caused some machinery to be out of sync. 14k in damages, I was 19 at the time. The boss just remarked I won't make the same mistake twice :) Always been cautious about the jobs I accept. Never anything medical again either.
Bastard Programmer from Hell :suss: "If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
-
Back when I worked for an aircraft company I was responsible for coding a set of traffic lights in the telemetry/testing office they were mounted upside down with green at the top. If certain things went over certain limits I was to make the amber light come on, further exceedance would cause the red light to come on at which point the aircraft controller would yell, "Eject, eject, eject", over the radio and the pilot would use the ejection seat to bail out before the multi-million dollar plane exploded. Lots of testing ensured the thing would work (feeding fake data from a recorded wire). I felt this was an enormous responsibility for a young (at the time) programmer to assume, especially since they had no peer reviewing mechanism or anything other than testing done by oneself. A life and lots of expensive equipment, not to mention where the bits crashed/fell were down to me getting it right. It was used once (correctly) while I was on vacation and twice more after I left the company; so all good. In a later job I programmed a software/hardware automation system. We nearly took a contract monitoring a nuclear power plant but the president of the company objected to the cost of the insurance policy we had to take as part of the deal and backed out. I was pleased about this although I had faith in our system I was happy to NOT have that responsibility. Has anyone else been in a similar situation?
- I would love to change the world, but they won’t give me the source code.
Not I. The closest I've been is to work on a commercial desktop program that keeps companies' escrow accounts balanced. If their escrow accounts can't balance, someone is going to jail. The risk is not even close to the same level as your project, but some people act as if it was. I tell all the new devs that work on the escrow account code, "If the money isn't right, then you don't have a job - no excuses."
Bond Keep all things as simple as possible, but no simpler. -said someone, somewhere
-
I work in automation systems so it's a fairly common-place thing for me to have a lot of scary responsibility of many varieties. Probably the most scary had to do a financial responsibility. I was working on a semiconductor processing system that was dealing with the chemical processing of wafers. These systems were very, very time-sensitive. We had several cassettes, each full of fifty wafers in process at a time in the system. One type of system dealt with acids that would etch off oxide that had been deposited on the wafers. Most customers would dilute the acid so the etching took longer but it was rather tolerant to immersion time variances. Other customers insisted on very high acid concentration which etched very fast and were very sensitive to the immersion time. It had to be accurate to the second. If the time was off by more than a second then the wafers over or under etched and the entire cassette was scrapped which resulted in the loss of millions of dollars of product. We spent an immense amount of time making sure that the entire process was exactly accurate. Imagine my frustration at how poorly the customer trained their operators. I had already been instructed to assume a sixth grade reading level from the operators and at times that seemed optimistic. I remember vividly watching an operator and guiding them during the first shift of production. We had to use a flaky wafer transfer unit that would load the cassettes into the system, as required by the customer, and these things would give errors often. When that happened the operator was supposed to cancel the transfer in progress and try again. Instead, I watched the operator press abort three times in the first shift, causing approximately ten million dollars in scrapped product. I was then instructed to put a double level of "are you sure" confirmations instead of just the one. I then had to tell them not to even use abort except for very special circumstances like power loss when the UPS couldn't keep the robots alive because the batteries were too weak. That was very, very frustrating. One amusing thing happened when we were installing a system in Erfurt, Thuringia, Germany. We were undergoing our final confirmation testing and the process engineer told us the etch times needed to be accurate to within one second. He emphasized this by saying, "and I mean German seconds, not American seconds." I got a good chuckle from that. When they approved the system he gave us a stopwatch that had been made in the USSR since Thur
I added "who did what when logging"; which everyone is aware of and can access (read). It does reduce the times they reach for the kill switch. And voice with the text: WARNING! WARNING! ... (and cross and skull bones here and there).
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it. ― Confucian Analects: Rules of Confucius about his food
-
I added "who did what when logging"; which everyone is aware of and can access (read). It does reduce the times they reach for the kill switch. And voice with the text: WARNING! WARNING! ... (and cross and skull bones here and there).
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it. ― Confucian Analects: Rules of Confucius about his food
I did that too with logged events but they couldn't be bothered to have separate accounts for each user so everyone used a generic "operator" account.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
-
Back when I worked for an aircraft company I was responsible for coding a set of traffic lights in the telemetry/testing office they were mounted upside down with green at the top. If certain things went over certain limits I was to make the amber light come on, further exceedance would cause the red light to come on at which point the aircraft controller would yell, "Eject, eject, eject", over the radio and the pilot would use the ejection seat to bail out before the multi-million dollar plane exploded. Lots of testing ensured the thing would work (feeding fake data from a recorded wire). I felt this was an enormous responsibility for a young (at the time) programmer to assume, especially since they had no peer reviewing mechanism or anything other than testing done by oneself. A life and lots of expensive equipment, not to mention where the bits crashed/fell were down to me getting it right. It was used once (correctly) while I was on vacation and twice more after I left the company; so all good. In a later job I programmed a software/hardware automation system. We nearly took a contract monitoring a nuclear power plant but the president of the company objected to the cost of the insurance policy we had to take as part of the deal and backed out. I was pleased about this although I had faith in our system I was happy to NOT have that responsibility. Has anyone else been in a similar situation?
- I would love to change the world, but they won’t give me the source code.
Some years ago I created the software to monitor the jacks under an aircraft as it was being serviced. Basically you could put load x on jacks y to z and unzip the plane around the middle without it falling apart, that kind of stuff. The actual, aircraft is no longer flying but it was that one with the long pointy nose that flew at Mach 2 (no names, no pack drill!). The whole project was being run by an engineering company so we were basically a sub contractor, the software was just down to me and it communicated with a "black box" that monitored the load cells on the jacks. We used an IBM PS2 computer which had been recently released and had a decent hi res graphics screen. The program had a configuration part where you entered the calibration information for each of the 15 or so load cells. When running it showed the load from each cell dynamically overlaid on an outline of the aircraft. Should a load exceed a preconfigured limit then a siren attached to the console would sound. I also supplied a detailed manual that covered all aspects of configuration and use. So, the main contractor picked up the whole caboodle and took it off the the airport for delivery to the maintenance hangars, my business partner and I decided to visit too but when we got to the hangar they had not arrived. We went on into town for another appointment and called in again on the way back. We were told that the main contractor had been but there were problems so thay had taken everything away with them. Later that day, early evening I had a call from the manufacturers of the load cells. The whole system had been dropped off with them and they were testing it using their massive presses. They found the system was working 100% with significantly higher accuracy than the minimum requested. Apparently an engineer had been busy jacking up the aircraft during the test. The jacks were manual and he began to have problems, so he fitted a 2ft extension to the jack handle and carried on before realising that something was not right and the actual load must be way more than what was displayed. The main contractor had taken my manual, ripped off the covers and added their own but did not bother to read it. Each load cell had characteristics that were up to 100% apart so they had the wrong load cells aligned to the wrong calibration parameters thus the cells were reading way off. So, now you know why said aircraft had a bent droopy nose :-)
-
Back when I worked for an aircraft company I was responsible for coding a set of traffic lights in the telemetry/testing office they were mounted upside down with green at the top. If certain things went over certain limits I was to make the amber light come on, further exceedance would cause the red light to come on at which point the aircraft controller would yell, "Eject, eject, eject", over the radio and the pilot would use the ejection seat to bail out before the multi-million dollar plane exploded. Lots of testing ensured the thing would work (feeding fake data from a recorded wire). I felt this was an enormous responsibility for a young (at the time) programmer to assume, especially since they had no peer reviewing mechanism or anything other than testing done by oneself. A life and lots of expensive equipment, not to mention where the bits crashed/fell were down to me getting it right. It was used once (correctly) while I was on vacation and twice more after I left the company; so all good. In a later job I programmed a software/hardware automation system. We nearly took a contract monitoring a nuclear power plant but the president of the company objected to the cost of the insurance policy we had to take as part of the deal and backed out. I was pleased about this although I had faith in our system I was happy to NOT have that responsibility. Has anyone else been in a similar situation?
- I would love to change the world, but they won’t give me the source code.
Back in the days when I worked for Orange Mobile Communications UK as an engineer, the department handled a lot of the call routing for the Tetra Network in the north of England and Scotland. On top of that we where also responsible for the B2B SMS routing systems used by a number of the large Metropolitan Scottish Fire services covering places like Fife, Glasgow and even Aberdeen. Aberdeenshire was of particular concern, because should an incident happen on one of the North Sea oil/gas rigs, Aberdeen would be the central operations area co-ordinating emergency services and evacuations. Tetra for those that don't know is the 2 way GSM based digital radio system that the UK police use, or did, I don't know if it's still used today :-) Basically though, our engineering team where responsible for making sure all those communications and SMS messages got sent to the correct message displays in the cab's of Fire Engines and Ambulances, and that police officers requests for back up and assistance where routed to the correct control rooms. One small slip up in the routing software or the systems controlling any of the GSM infrastructure could result in a Fire Engine going to the Wrong place, or some one needing help not getting it in time, needless to say we where very quick to investigate even the smallest most innocuous report for any of those systems.
-
Back when I worked for an aircraft company I was responsible for coding a set of traffic lights in the telemetry/testing office they were mounted upside down with green at the top. If certain things went over certain limits I was to make the amber light come on, further exceedance would cause the red light to come on at which point the aircraft controller would yell, "Eject, eject, eject", over the radio and the pilot would use the ejection seat to bail out before the multi-million dollar plane exploded. Lots of testing ensured the thing would work (feeding fake data from a recorded wire). I felt this was an enormous responsibility for a young (at the time) programmer to assume, especially since they had no peer reviewing mechanism or anything other than testing done by oneself. A life and lots of expensive equipment, not to mention where the bits crashed/fell were down to me getting it right. It was used once (correctly) while I was on vacation and twice more after I left the company; so all good. In a later job I programmed a software/hardware automation system. We nearly took a contract monitoring a nuclear power plant but the president of the company objected to the cost of the insurance policy we had to take as part of the deal and backed out. I was pleased about this although I had faith in our system I was happy to NOT have that responsibility. Has anyone else been in a similar situation?
- I would love to change the world, but they won’t give me the source code.
While I have developed "scary" systems, the scariest was being a victim of faulty programming in my Ford F150. It seems that Ford had gone to what they call EPAS--Electronically Powered Assisted Steering in my 2012 F150. Essentially, EPAS is drive by wire and has nifty software features such trailer sway control and crown load detection. In one circumstance, I was driving a country road, hit a significant deflection in the road surface and the truck took over steering and rolled. As far as I can guess, it seems that there the software had an algorithm for "roll-over protection" and it "thought" the truck was rolling. But instead of correcting by simply holding straight, it elected to counter-steer. I ended up hanging upside down from my seat belt while the truck dialed 911 and got an emergency dispatcher on the line. Truck was considered totaled because the airbags had deployed and it costs more to "clean"the truck and replace the airbags than the truck was worth. The irony is that about three weeks after I settled with my insurance company, I got a urgent recall notice from Ford to bring my truck in and have the software updated. I will not trust drive-by-wire in difficult situations---No software developer or engineer can encode ALL the circumstances that might be encountered.
-
Not me, but a relative of mine: He did his (compulsory) service in the military at a time when computing technology consisted of a slide rule[^] and a book of tables. Having an engineering education, he was given responsibility for calculating the launch angle of the mortars. Gradually, he realized that if he made a miscalculation, that could lead to several young men returning to their wives, their kids and parents alive. If he did his task properly, they might return home, but in a coffin. Parents would loose their sons, wives their husbands, kids their parents. Even soldiers are humans, with a life. Even enemy soldiers. He couldn't take that responsibility: Making a mistake saves lives. Doing the job correctly kills humans. Sitting back home, or in a nuclear proof shelter, it is easy to say: But this is war. It is different when you are personally responsible. You could do a deliberate miscalculation, to let people live. He went to the military administration telling that he refused to take that on himself. He refused further military service. In those days, that was like openly branding yourself as An Enemy of The People. Your relatives and friends might turn their back on you. Yet, he chose that, rather than knowing that his correct calculations would cause people to loose their loved ones. Making himself a murderer, disguised in a military uniform. Of course we all know that the commandment "Thou shalt not kill" really is a short form of "Thou shalt not kill any of us". Many of us live by the extended version. Maybe you could strangle your neighbor with your bare hands, or exterminate him with a mortar, because of his religion, his ideas about economy, or some moral ideas he might have. That's what usually identify the 'enemy' that must be fought. And killed, if he does not submit to your religious, economic and moral ideas. He couldn't. Nor could I. But in my days of youth, requesting for a non-military 16 months service was far more accepted. I never had to flee from a personal responsibility comparable to that of this relative of mine.
I'm just now catching up with a small backlog of posts I wanted to go back to, and stumbled upon this.
trønderen wrote:
he realized that if he made a miscalculation, that could lead to several young men returning to their wives, their kids and parents alive. If he did his task properly, they might return home, but in a coffin.
Interesting conundrum, and I understand his position... But OTOH, it's not like sparing the enemy means lives are saved and that's the end of the story...rather, if you don't kill your enemy, he gets a chance to kill you, your family, and your follow countrymen. Is that a better outcome? If it's kill or be killed, it's a pretty simple decision in my mind. But ultimately, when you've reached this point, when wars are waged, invariably it's because some "leader" has failed miserably at his job. I didn't really mean to revive this old thread, but somehow I felt compelled to contribute my 2 cents. :-)