Safety critical software
-
Those of us involved in commercial software development pretty much know that we take shortcuts sometimes, that documentation isn't always what it should be, security not quite 100% etc. Hey, these are all influenced by financial constraints. But when it comes to safety-critical stuff (nuclear power stations, air traffic control, railway signalling) I'd rather hoped that stuff was done properly. As a railway enthusiast, I always read reports from the UK's Rail Accident Investigation Branch. It can be a sombre illustration of how "silly little things" can add up to a catastrophic outcome. As a volunteer on a heritage railway operation, any reminder of the potential significance of our actions on the safety of the railway is always welcomed. Today the RAIB has released a report into the loss of speed restriction info on a line in Wales. May not sound serious, but temporary speed restrictions, relayed wirelessly to train drivers, may be there to slow a train passing over a track defect; at full line speed a serious accident could be the result. The report - at https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/749430/IR012018_181018_Cambrian_TSRs.pdf[^] is an interim one pending further investigation. Essentially, some data did not get transferred from one system to another following a software reset. Then neither system identified the data was missing. Then the trains did not report they'd not received the data. When the problem was found, local support staff deleted the "corrupt" data and the system logfiles, before reloading everything, so there's no diagnostic data. Then the original authors of the software, originally designed for a high-speed line in France, had to reverse-engineer the software to find out "how it was designed to work" (err, no, they'll find out how it was implemented, not what it was designed to do!) Plus much of the software, including the system name acronyms, are in French ("poste de GEstion des Signalisations Temporaires" = GEST system) and train speeds and restrictions are in Km/h - despite the rest of the UK railway system, as per its roads, being expressed in miles per hour, AND all railway distances
My step-son is a train driver and his wife is a signaller. I would trust both of them far more than any foreign made system. Maybe your message should go to the Ministry of Transport, or whatever current cowboy outfit is responsible for the rail network in this country.
-
Those of us involved in commercial software development pretty much know that we take shortcuts sometimes, that documentation isn't always what it should be, security not quite 100% etc. Hey, these are all influenced by financial constraints. But when it comes to safety-critical stuff (nuclear power stations, air traffic control, railway signalling) I'd rather hoped that stuff was done properly. As a railway enthusiast, I always read reports from the UK's Rail Accident Investigation Branch. It can be a sombre illustration of how "silly little things" can add up to a catastrophic outcome. As a volunteer on a heritage railway operation, any reminder of the potential significance of our actions on the safety of the railway is always welcomed. Today the RAIB has released a report into the loss of speed restriction info on a line in Wales. May not sound serious, but temporary speed restrictions, relayed wirelessly to train drivers, may be there to slow a train passing over a track defect; at full line speed a serious accident could be the result. The report - at https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/749430/IR012018_181018_Cambrian_TSRs.pdf[^] is an interim one pending further investigation. Essentially, some data did not get transferred from one system to another following a software reset. Then neither system identified the data was missing. Then the trains did not report they'd not received the data. When the problem was found, local support staff deleted the "corrupt" data and the system logfiles, before reloading everything, so there's no diagnostic data. Then the original authors of the software, originally designed for a high-speed line in France, had to reverse-engineer the software to find out "how it was designed to work" (err, no, they'll find out how it was implemented, not what it was designed to do!) Plus much of the software, including the system name acronyms, are in French ("poste de GEstion des Signalisations Temporaires" = GEST system) and train speeds and restrictions are in Km/h - despite the rest of the UK railway system, as per its roads, being expressed in miles per hour, AND all railway distances
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
-
My step-son is a train driver and his wife is a signaller. I would trust both of them far more than any foreign made system. Maybe your message should go to the Ministry of Transport, or whatever current cowboy outfit is responsible for the rail network in this country.
Fortunately RAIB are investigating further, and have the nous to identify just basic stuff like the above. It's not rocket science to stop 1st level support being able to delete the logfiles, surely. Hopefully they will dig a bit deeper (my comments above based purely on their interim summary) and knock a little sense into the industry.
-
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
Quite. And as I alluded to in the post, I know (to an extent) when I have taken shortcuts or made assumptions. However clients don't have unlimited budgets and I have to balance costs against benefits. The difference is that in most commercial systems, the "benefits" are a few quid on the bottom line. For safety critical systems, the "benefits" are that large numbers of people don't get killed. Hopefully that tips the cost/benefit balance a little further and justifies some measures and controls that otherwise it wouldn't be worth implementing; like at the start of a data transfer, telling the receiving system how many files to expect and their total size. Checking the creation dates might be handy, too!
-
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
DRHuff wrote:
Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
I've (thankfully) never written software that people's lives have depended on. If I worked in such an industry I don't doubt the testing and requirement verification is on a higher level. We dealt with this a little in university and were taught how "formal methods" is used to verify safety-critical things. Formal methods - Wikipedia[^] The worse my code ever did was cancel a few people's credit cards :-O but that's a story for a different thread :)
-
DRHuff wrote:
Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
I've (thankfully) never written software that people's lives have depended on. If I worked in such an industry I don't doubt the testing and requirement verification is on a higher level. We dealt with this a little in university and were taught how "formal methods" is used to verify safety-critical things. Formal methods - Wikipedia[^] The worse my code ever did was cancel a few people's credit cards :-O but that's a story for a different thread :)
F-ES Sitecore wrote:
I've (thankfully) never written software that people's lives have depended on.
My brother works as a civilian for the air force here in the states. he is part of the software test team for jet fighters. That's all I know, and that is all he could tell me. At a high level, all software written for jet fighters (weapons systems, diagnostics, live data feeds, etc.) all has to be tested, and he is one of the lead testers. That would be too much stress for me. Jet fighter's life depends on good testing, and anyone on the ground (civs and soldiers, etc.) depend on good testing. No thank you.
-
F-ES Sitecore wrote:
I've (thankfully) never written software that people's lives have depended on.
My brother works as a civilian for the air force here in the states. he is part of the software test team for jet fighters. That's all I know, and that is all he could tell me. At a high level, all software written for jet fighters (weapons systems, diagnostics, live data feeds, etc.) all has to be tested, and he is one of the lead testers. That would be too much stress for me. Jet fighter's life depends on good testing, and anyone on the ground (civs and soldiers, etc.) depend on good testing. No thank you.
When working for [name redacted] jet fighter development team in the UK back in the 1980's, I was responsible for the upside down "traffic lights" in the telemetry room (green at the top, red at the bottom). If the green light was on then everything was fine, amber light when certain readings were close to being above or below safe values, ...and when the red light came on the flight controller only said three words, "Eject, Eject, Eject!" It definitely freaked a few people out when I was testing using tweaked recorded data! I don't think I ever reviewed my code more than on that project. Weirdly, nobody else checked my code. I was the only programmer on that project with just a couple of aero-engineers to provide advice and the appropriate numbers. I only saw it happen once during an actual test flight in the next four years after I wrote it. :omg: I was in the telemetry room at the time and it really freaked me out! The monitoring engineers reacted so calmly you would have thought someone had just sneezed or something. It was, "Eject, eject, eject", then a bang and a cut-off whoosh from the pilot's r/t, then a report from the chase plane about the safe exit of the test-pilot :wtf: and the jet crashing into the sea. :sigh: No-one asked me whether my software was correct or not; they just assumed it was - and wrote off a multi-million dollar jet-fighter prototype without any further input from me. The system remained in place for nearly 12 additional years after I left and, apparently, was activated three or four more times.
- I would love to change the world, but they won’t give me the source code.
-
When working for [name redacted] jet fighter development team in the UK back in the 1980's, I was responsible for the upside down "traffic lights" in the telemetry room (green at the top, red at the bottom). If the green light was on then everything was fine, amber light when certain readings were close to being above or below safe values, ...and when the red light came on the flight controller only said three words, "Eject, Eject, Eject!" It definitely freaked a few people out when I was testing using tweaked recorded data! I don't think I ever reviewed my code more than on that project. Weirdly, nobody else checked my code. I was the only programmer on that project with just a couple of aero-engineers to provide advice and the appropriate numbers. I only saw it happen once during an actual test flight in the next four years after I wrote it. :omg: I was in the telemetry room at the time and it really freaked me out! The monitoring engineers reacted so calmly you would have thought someone had just sneezed or something. It was, "Eject, eject, eject", then a bang and a cut-off whoosh from the pilot's r/t, then a report from the chase plane about the safe exit of the test-pilot :wtf: and the jet crashing into the sea. :sigh: No-one asked me whether my software was correct or not; they just assumed it was - and wrote off a multi-million dollar jet-fighter prototype without any further input from me. The system remained in place for nearly 12 additional years after I left and, apparently, was activated three or four more times.
- I would love to change the world, but they won’t give me the source code.
Very interesting story. That is actually kind of cool that you got to work on a project like that. I am fairly certain, that I would have soiled myself, if my software was used in an "Eject, eject, eject!" scenario, even if it worked as expected.
-
When working for [name redacted] jet fighter development team in the UK back in the 1980's, I was responsible for the upside down "traffic lights" in the telemetry room (green at the top, red at the bottom). If the green light was on then everything was fine, amber light when certain readings were close to being above or below safe values, ...and when the red light came on the flight controller only said three words, "Eject, Eject, Eject!" It definitely freaked a few people out when I was testing using tweaked recorded data! I don't think I ever reviewed my code more than on that project. Weirdly, nobody else checked my code. I was the only programmer on that project with just a couple of aero-engineers to provide advice and the appropriate numbers. I only saw it happen once during an actual test flight in the next four years after I wrote it. :omg: I was in the telemetry room at the time and it really freaked me out! The monitoring engineers reacted so calmly you would have thought someone had just sneezed or something. It was, "Eject, eject, eject", then a bang and a cut-off whoosh from the pilot's r/t, then a report from the chase plane about the safe exit of the test-pilot :wtf: and the jet crashing into the sea. :sigh: No-one asked me whether my software was correct or not; they just assumed it was - and wrote off a multi-million dollar jet-fighter prototype without any further input from me. The system remained in place for nearly 12 additional years after I left and, apparently, was activated three or four more times.
- I would love to change the world, but they won’t give me the source code.
You should definitely tell us when you write your biography. :)
Wrong is evil and must be defeated. - Jeff Ello
-
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
Back in the '90s I had some involvement in safety critical hardware and (less so) software. I did a lot of work looking at potential failures of the hardware, including IC pins stuck at 0, 1, open circuit, shorted, and what would happen if a 5mm piece of swarf happened to short between two pins. Taking the drawings for PCBs and then putting 5mm radius circles around every pin was a delightful experience. For the four boards I worked on, I produced a pile of paper about 10" high. The requirement was that no single failure could cause loss of mission. Everything was dual channel and the loss of a channel was OK to an extent. I didn't work on it, by my company at the time did write the software to control a plane (probably many more since). Took me more than 20 years before I plucked up the courage to fly on one such make of plane.
-
Reminds me of the story Scott Meyers related at a conference. He asked a question of a software audience - "If your company wrote the flight control software for the plane you flew on to come to this conference - how many of you would feel safe and have got on the plane?" One guy in the audience was the only one to hold up his hand. Scott asked him "So you would feel safe getting onto a plane if your company wrote the software?" The reply was - "If my company wrote the software I am sure that the plane wouldn't be able to move away from the gate!" Think of all the places you have worked for writing software and imagine the quality of safety critical systems that they might have produced. Now do you think it is significantly different any where else? Think of all the specs you have written or read. Think of how many didn't foresee subtle problems that cropped up only late in development or testing and only appeared in a rare set of circumstances. Do you think that you caught all the rare sets of circumstances that can happen? Did you foresee all the possible subtle problems? Are you sure?
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
DRHuff wrote:
Now do you think it is significantly different any where else
Yes, definitely. I write safety critical software, or better said am overall responsible for its specifications and releases, and we have process requirements concerning reviews and validation as well as norms to fullfill that you cannot imagine exist. The SW is of course not bullet proof, even with those, but quality is by light-years better than anything I have ever come across in another development context. Will it stay so ? This mostly relies on the fact that HW costs money, so the HW specs are kept small. Therefor you have to otpimize SW and need really skilled people to program, limited resources so no fancy library or frameworks or whatever, and acceptable requirements, since also the customer knows that not everything is possible on a micro controller. Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
-
You should definitely tell us when you write your biography. :)
Wrong is evil and must be defeated. - Jeff Ello
For some parts of the Official Secrets Act there is a 30-year limit, for others, a 90-year wait. The 30-years expired in 2017 but I need to check on the 90-year limits before I set pen to paper. In another 60 years I might be too old to remember all the juicy details (or care)!
- I would love to change the world, but they won’t give me the source code.
-
DRHuff wrote:
Now do you think it is significantly different any where else
Yes, definitely. I write safety critical software, or better said am overall responsible for its specifications and releases, and we have process requirements concerning reviews and validation as well as norms to fullfill that you cannot imagine exist. The SW is of course not bullet proof, even with those, but quality is by light-years better than anything I have ever come across in another development context. Will it stay so ? This mostly relies on the fact that HW costs money, so the HW specs are kept small. Therefor you have to otpimize SW and need really skilled people to program, limited resources so no fancy library or frameworks or whatever, and acceptable requirements, since also the customer knows that not everything is possible on a micro controller. Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
Rage wrote:
Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
I went to a session on embedded programming at a conference a couple of years ago. The example being show cased was a small wheeled robot controlled by a computer. I was dissappointed when I found out that it was a high end Wi-Fi capable Arduino compatible board running a NodeJS server, and the control client on the laptop sent instructions to the robot as HTTP requests.
-
Rage wrote:
Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
I went to a session on embedded programming at a conference a couple of years ago. The example being show cased was a small wheeled robot controlled by a computer. I was dissappointed when I found out that it was a high end Wi-Fi capable Arduino compatible board running a NodeJS server, and the control client on the laptop sent instructions to the robot as HTTP requests.
Exactly this. In my case (automotive), I see more and more high-end computers ocming, especially for autonomous driving. :~
-
Exactly this. In my case (automotive), I see more and more high-end computers ocming, especially for autonomous driving. :~
You won't catch me driving (or riding in?) one of those things. I'm nervous enough about the fact that my brake and accelerator pedals are analogue switches feeding into a computer. I don't want my life depending on a computer that only requires my level of skill to program.
-
For some parts of the Official Secrets Act there is a 30-year limit, for others, a 90-year wait. The 30-years expired in 2017 but I need to check on the 90-year limits before I set pen to paper. In another 60 years I might be too old to remember all the juicy details (or care)!
- I would love to change the world, but they won’t give me the source code.
So, what are you waiting for? :)
Wrong is evil and must be defeated. - Jeff Ello
-
So, what are you waiting for? :)
Wrong is evil and must be defeated. - Jeff Ello
-
DRHuff wrote:
Now do you think it is significantly different any where else
Yes, definitely. I write safety critical software, or better said am overall responsible for its specifications and releases, and we have process requirements concerning reviews and validation as well as norms to fullfill that you cannot imagine exist. The SW is of course not bullet proof, even with those, but quality is by light-years better than anything I have ever come across in another development context. Will it stay so ? This mostly relies on the fact that HW costs money, so the HW specs are kept small. Therefor you have to otpimize SW and need really skilled people to program, limited resources so no fancy library or frameworks or whatever, and acceptable requirements, since also the customer knows that not everything is possible on a micro controller. Now more and more embedded systems get bigger computers, with sometimes even ... java-based systems, or embedded windows X| so not sure it will go on.
I am aware that it is different (vastly different actually) for writing safety critical systems. And that it is possible to write relatively bullet proof software. But it requires a discipline and structure that most dev houses do not have. I was talking to the head of development for an avionics GPS receiver and he told me the test document when printed was approaching 8 feet of paper - and they weren't done yet. I took a course on the DO178 avionics software specification and quickly concluded that my little 3 man shop was never going to put our device into a helicopter as anything but a 'temporary install'. Everything had to be 178 approved - which left you with bare metal or one severely limited version of Linux that cost a fortune to run on as an OS. No dead code, a test suite that can execute any given line of code, etc. That was an interesting course! And I will never build anything for helicopters again! Where a huge part of the problem lies is in the things you don't see. The difficulty in writing a spec that catches all the edge cases, fault tolerances, etc is usually revealed by the post accident case studies that show why something happened and nobody ever caught it because nobody ever thought it up in the first place. It may be obvious post flight analysis that the acceleration variables type size is too small to hold the value generated by a bigger booster rocket - but good luck finding it before you launch! (Although I thought that one should have been caught beforehand in simulation, but...)
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
-
I am aware that it is different (vastly different actually) for writing safety critical systems. And that it is possible to write relatively bullet proof software. But it requires a discipline and structure that most dev houses do not have. I was talking to the head of development for an avionics GPS receiver and he told me the test document when printed was approaching 8 feet of paper - and they weren't done yet. I took a course on the DO178 avionics software specification and quickly concluded that my little 3 man shop was never going to put our device into a helicopter as anything but a 'temporary install'. Everything had to be 178 approved - which left you with bare metal or one severely limited version of Linux that cost a fortune to run on as an OS. No dead code, a test suite that can execute any given line of code, etc. That was an interesting course! And I will never build anything for helicopters again! Where a huge part of the problem lies is in the things you don't see. The difficulty in writing a spec that catches all the edge cases, fault tolerances, etc is usually revealed by the post accident case studies that show why something happened and nobody ever caught it because nobody ever thought it up in the first place. It may be obvious post flight analysis that the acceleration variables type size is too small to hold the value generated by a bigger booster rocket - but good luck finding it before you launch! (Although I thought that one should have been caught beforehand in simulation, but...)
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
DRHuff wrote:
is usually revealed by the post accident case studies
Yep, this is the problem. We cannot test à la Microsoft = Release the half-finished product and wait for the users complains to identify the bugs. We have to use several systematical methods for analysing requirements and use-cases, but I said it already, it is not 100% and will never be. It gets even more complicated now, because even the less "important" embedded ECUs are concerned by security and safety issues : Cars will have OTA update capabilities for most of their onboard ECUs. Now hack this and get the steering or the gasoline pump ECU to shutdown - millions of car getting the update and instantly blocked on the road. Because of a gasoline pump software, probably the smallest piece of code in the car. :~
-
DRHuff wrote:
is usually revealed by the post accident case studies
Yep, this is the problem. We cannot test à la Microsoft = Release the half-finished product and wait for the users complains to identify the bugs. We have to use several systematical methods for analysing requirements and use-cases, but I said it already, it is not 100% and will never be. It gets even more complicated now, because even the less "important" embedded ECUs are concerned by security and safety issues : Cars will have OTA update capabilities for most of their onboard ECUs. Now hack this and get the steering or the gasoline pump ECU to shutdown - millions of car getting the update and instantly blocked on the road. Because of a gasoline pump software, probably the smallest piece of code in the car. :~
Yep - I really don't think that the IoT will work out as well as everybody hopes. Not by a long shot. I think that OTA updates for cars will quickly become a thing of the past as the hacking danger becomes too great. You will have to get it updated at the dealership - for the low low service fee of $119.99!
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)