Safety critical software
-
When working for [name redacted] jet fighter development team in the UK back in the 1980's, I was responsible for the upside down "traffic lights" in the telemetry room (green at the top, red at the bottom). If the green light was on then everything was fine, amber light when certain readings were close to being above or below safe values, ...and when the red light came on the flight controller only said three words, "Eject, Eject, Eject!" It definitely freaked a few people out when I was testing using tweaked recorded data! I don't think I ever reviewed my code more than on that project. Weirdly, nobody else checked my code. I was the only programmer on that project with just a couple of aero-engineers to provide advice and the appropriate numbers. I only saw it happen once during an actual test flight in the next four years after I wrote it. :omg: I was in the telemetry room at the time and it really freaked me out! The monitoring engineers reacted so calmly you would have thought someone had just sneezed or something. It was, "Eject, eject, eject", then a bang and a cut-off whoosh from the pilot's r/t, then a report from the chase plane about the safe exit of the test-pilot :wtf: and the jet crashing into the sea. :sigh: No-one asked me whether my software was correct or not; they just assumed it was - and wrote off a multi-million dollar jet-fighter prototype without any further input from me. The system remained in place for nearly 12 additional years after I left and, apparently, was activated three or four more times.
- I would love to change the world, but they won’t give me the source code.
Very interesting story. That is actually kind of cool that you got to work on a project like that. I am fairly certain, that I would have soiled myself, if my software was used in an "Eject, eject, eject!" scenario, even if it worked as expected.
-
When working for [name redacted] jet fighter development team in the UK back in the 1980's, I was responsible for the upside down "traffic lights" in the telemetry room (green at the top, red at the bottom). If the green light was on then everything was fine, amber light when certain readings were close to being above or below safe values, ...and when the red light came on the flight controller only said three words, "Eject, Eject, Eject!" It definitely freaked a few people out when I was testing using tweaked recorded data! I don't think I ever reviewed my code more than on that project. Weirdly, nobody else checked my code. I was the only programmer on that project with just a couple of aero-engineers to provide advice and the appropriate numbers. I only saw it happen once during an actual test flight in the next four years after I wrote it. :omg: I was in the telemetry room at the time and it really freaked me out! The monitoring engineers reacted so calmly you would have thought someone had just sneezed or something. It was, "Eject, eject, eject", then a bang and a cut-off whoosh from the pilot's r/t, then a report from the chase plane about the safe exit of the test-pilot :wtf: and the jet crashing into the sea. :sigh: No-one asked me whether my software was correct or not; they just assumed it was - and wrote off a multi-million dollar jet-fighter prototype without any further input from me. The system remained in place for nearly 12 additional years after I left and, apparently, was activated three or four more times.
- I would love to change the world, but they won’t give me the source code.
You should definitely tell us when you write your biography. :)
Wrong is evil and must be defeated. - Jeff Ello
-
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
Back in the '90s I had some involvement in safety critical hardware and (less so) software. I did a lot of work looking at potential failures of the hardware, including IC pins stuck at 0, 1, open circuit, shorted, and what would happen if a 5mm piece of swarf happened to short between two pins. Taking the drawings for PCBs and then putting 5mm radius circles around every pin was a delightful experience. For the four boards I worked on, I produced a pile of paper about 10" high. The requirement was that no single failure could cause loss of mission. Everything was dual channel and the loss of a channel was OK to an extent. I didn't work on it, by my company at the time did write the software to control a plane (probably many more since). Took me more than 20 years before I plucked up the courage to fly on one such make of plane.
-
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
DRHuff wrote:
Now do you think it is significantly different any where else
Yes, definitely. I write safety critical software, or better said am overall responsible for its specifications and releases, and we have process requirements concerning reviews and validation as well as norms to fullfill that you cannot imagine exist. The SW is of course not bullet proof, even with those, but quality is by light-years better than anything I have ever come across in another development context. Will it stay so ? This mostly relies on the fact that HW costs money, so the HW specs are kept small. Therefor you have to otpimize SW and need really skilled people to program, limited resources so no fancy library or frameworks or whatever, and acceptable requirements, since also the customer knows that not everything is possible on a micro controller. Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
-
You should definitely tell us when you write your biography. :)
Wrong is evil and must be defeated. - Jeff Ello
For some parts of the Official Secrets Act there is a 30-year limit, for others, a 90-year wait. The 30-years expired in 2017 but I need to check on the 90-year limits before I set pen to paper. In another 60 years I might be too old to remember all the juicy details (or care)!
- I would love to change the world, but they won’t give me the source code.
-
DRHuff wrote:
Now do you think it is significantly different any where else
Yes, definitely. I write safety critical software, or better said am overall responsible for its specifications and releases, and we have process requirements concerning reviews and validation as well as norms to fullfill that you cannot imagine exist. The SW is of course not bullet proof, even with those, but quality is by light-years better than anything I have ever come across in another development context. Will it stay so ? This mostly relies on the fact that HW costs money, so the HW specs are kept small. Therefor you have to otpimize SW and need really skilled people to program, limited resources so no fancy library or frameworks or whatever, and acceptable requirements, since also the customer knows that not everything is possible on a micro controller. Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
Rage wrote:
Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
I went to a session on embedded programming at a conference a couple of years ago. The example being show cased was a small wheeled robot controlled by a computer. I was dissappointed when I found out that it was a high end Wi-Fi capable Arduino compatible board running a NodeJS server, and the control client on the laptop sent instructions to the robot as HTTP requests.
-
Rage wrote:
Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
I went to a session on embedded programming at a conference a couple of years ago. The example being show cased was a small wheeled robot controlled by a computer. I was dissappointed when I found out that it was a high end Wi-Fi capable Arduino compatible board running a NodeJS server, and the control client on the laptop sent instructions to the robot as HTTP requests.
Exactly this. In my case (automotive), I see more and more high-end computers ocming, especially for autonomous driving. :~
-
Exactly this. In my case (automotive), I see more and more high-end computers ocming, especially for autonomous driving. :~
You won't catch me driving (or riding in?) one of those things. I'm nervous enough about the fact that my brake and accelerator pedals are analogue switches feeding into a computer. I don't want my life depending on a computer that only requires my level of skill to program.
-
For some parts of the Official Secrets Act there is a 30-year limit, for others, a 90-year wait. The 30-years expired in 2017 but I need to check on the 90-year limits before I set pen to paper. In another 60 years I might be too old to remember all the juicy details (or care)!
- I would love to change the world, but they won’t give me the source code.
So, what are you waiting for? :)
Wrong is evil and must be defeated. - Jeff Ello
-
So, what are you waiting for? :)
Wrong is evil and must be defeated. - Jeff Ello
-
DRHuff wrote:
Now do you think it is significantly different any where else
Yes, definitely. I write safety critical software, or better said am overall responsible for its specifications and releases, and we have process requirements concerning reviews and validation as well as norms to fullfill that you cannot imagine exist. The SW is of course not bullet proof, even with those, but quality is by light-years better than anything I have ever come across in another development context. Will it stay so ? This mostly relies on the fact that HW costs money, so the HW specs are kept small. Therefor you have to otpimize SW and need really skilled people to program, limited resources so no fancy library or frameworks or whatever, and acceptable requirements, since also the customer knows that not everything is possible on a micro controller. Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
I am aware that it is different (vastly different actually) for writing safety critical systems. And that it is possible to write relatively bullet proof software. But it requires a discipline and structure that most dev houses do not have. I was talking to the head of development for an avionics GPS receiver and he told me the test document when printed was approaching 8 feet of paper - and they weren't done yet. I took a course on the DO178 avionics software specification and quickly concluded that my little 3 man shop was never going to put our device into a helicopter as anything but a 'temporary install'. Everything had to be 178 approved - which left you with bare metal or one severely limited version of Linux that cost a fortune to run on as an OS. No dead code, a test suite that can execute any given line of code, etc. That was an interesting course! And I will never build anything for helicopters again! Where a huge part of the problem lies is in the things you don't see. The difficulty in writing a spec that catches all the edge cases, fault tolerances, etc is usually revealed by the post accident case studies that show why something happened and nobody ever caught it because nobody ever thought it up in the first place. It may be obvious post flight analysis that the acceleration variables type size is too small to hold the value generated by a bigger booster rocket - but good luck finding it before you launch! (Although I thought that one should have been caught beforehand in simulation, but...)
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
-
I am aware that it is different (vastly different actually) for writing safety critical systems. And that it is possible to write relatively bullet proof software. But it requires a discipline and structure that most dev houses do not have. I was talking to the head of development for an avionics GPS receiver and he told me the test document when printed was approaching 8 feet of paper - and they weren't done yet. I took a course on the DO178 avionics software specification and quickly concluded that my little 3 man shop was never going to put our device into a helicopter as anything but a 'temporary install'. Everything had to be 178 approved - which left you with bare metal or one severely limited version of Linux that cost a fortune to run on as an OS. No dead code, a test suite that can execute any given line of code, etc. That was an interesting course! And I will never build anything for helicopters again! Where a huge part of the problem lies is in the things you don't see. The difficulty in writing a spec that catches all the edge cases, fault tolerances, etc is usually revealed by the post accident case studies that show why something happened and nobody ever caught it because nobody ever thought it up in the first place. It may be obvious post flight analysis that the acceleration variables type size is too small to hold the value generated by a bigger booster rocket - but good luck finding it before you launch! (Although I thought that one should have been caught beforehand in simulation, but...)
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
DRHuff wrote:
is usually revealed by the post accident case studies
Yep, this is the problem. We cannot test à la Microsoft = Release the half-finished product and wait for the users complains to identify the bugs. We have to use several systematical methods for analysing requirements and use-cases, but I said it already, it is not 100% and will never be. It gets even more complicated now, because even the less "important" embedded ECUs are concerned by security and safety issues : Cars will have OTA update capabilities for most of their onboard ECUs. Now hack this and get the steering or the gasoline pump ECU to shutdown - millions of car getting the update and instantly blocked on the road. Because of a gasoline pump software, probably the smallest piece of code in the car. :~
-
DRHuff wrote:
is usually revealed by the post accident case studies
Yep, this is the problem. We cannot test à la Microsoft = Release the half-finished product and wait for the users complains to identify the bugs. We have to use several systematical methods for analysing requirements and use-cases, but I said it already, it is not 100% and will never be. It gets even more complicated now, because even the less "important" embedded ECUs are concerned by security and safety issues : Cars will have OTA update capabilities for most of their onboard ECUs. Now hack this and get the steering or the gasoline pump ECU to shutdown - millions of car getting the update and instantly blocked on the road. Because of a gasoline pump software, probably the smallest piece of code in the car. :~
Yep - I really don't think that the IoT will work out as well as everybody hopes. Not by a long shot. I think that OTA updates for cars will quickly become a thing of the past as the hacking danger becomes too great. You will have to get it updated at the dealership - for the low low service fee of $119.99!
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
-
Yep - I really don't think that the IoT will work out as well as everybody hopes. Not by a long shot. I think that OTA updates for cars will quickly become a thing of the past as the hacking danger becomes too great. You will have to get it updated at the dealership - for the low low service fee of $119.99!
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
DRHuff wrote:
the hacking danger becomes too great
Interestingly, I think it is the costs of ... SW that will limit the thing => We start to get requirements for encrypting our interfaces and the internal communication on the vehicle bus. From the hw point of view, on a small ECU with for instance 16kB RAM, you need ... 14kB for the encryption stuff, leaving you with 2kB for the functional SW :wtf: - this is not sufficient. So everybody must add RAM, which means PCB modifications, which means 0,1 to 0,5€ more for each part, and parts are counted in the range of millions to tens of millions for _each_ ECU in the vehicle. Currently, there must be around 200 ECUs of different sizes in one car. And this is without development costs. No customer is going to pay that for the mere sake of "security", unless "cybersafe" vehicles of the future are going to be sold 500k€ each to the end customer, which I very much doubt.
-
DRHuff wrote:
the hacking danger becomes too great
Interestingly, I think it is the costs of ... SW that will limit the thing => We start to get requirements for encrypting our interfaces and the internal communication on the vehicle bus. From the hw point of view, on a small ECU with for instance 16kB RAM, you need ... 14kB for the encryption stuff, leaving you with 2kB for the functional SW :wtf: - this is not sufficient. So everybody must add RAM, which means PCB modifications, which means 0,1 to 0,5€ more for each part, and parts are counted in the range of millions to tens of millions for _each_ ECU in the vehicle. Currently, there must be around 200 ECUs of different sizes in one car. And this is without development costs. No customer is going to pay that for the mere sake of "security", unless "cybersafe" vehicles of the future are going to be sold 500k€ each to the end customer, which I very much doubt.
Which is why I think that OTA updates will disappear. If the update has to happen while it is hardwired that greatly limits the opportunities for hackers to get into the system. Not sure I understand what you mean by "which means 0,1 to 0,5€ more for each part, and parts are counted in the range of millions to tens of millions for _each_ ECU in the vehicle" What is a 'part' vs an 'ECU'? While there are several hundred ECUs in the vehicle, increasing the RAM for security purposes might increase the cost per ECU by a buck or two. A couple of hundred dollars for a car that won't kill me because some script kiddie got into it would be worth it. BTW - thanks for the discussion - it's been good. (A good discussion on the internet - what's next - bug free software!)
-
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
DRHuff wrote:
Now do you think it is significantly different any where else?
No. And I am not talking about software - I mean any industry. People working in restaurants do stupid things. People building buildings do stupid things. Doctors do stupid things. Police do stupid things. Military does stupid things.
DRHuff wrote:
Did you foresee all the possible subtle problems? Are you sure?
No. And while I look for those sort of things I have only been doing it for about 15 years. And no one else at the companies that I worked for was doing that.