UK Track and Trace system
-
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
-
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
F-ES Sitecore wrote:
What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate.
Too good! :laugh:
Robust Services Core | Software Techniques for Lemmings | Articles
The fox knows many things, but the hedgehog knows one big thing. -
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
Quote:
on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate.
I didn't know they had Twitter on planet Vulcan. Not sure why a species that is guided by logic would need Twitter or even know how to use it, given here on planet Earth Twitter is used for misinformation, lies, social engineering, manipulation and emotional outbursts, but what do I know.
Latest Articles:
Thread Safe Quantized Temporal Frame Ring Buffer -
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
But ... but ... it's a World-Beating System! The government says so, and it wouldn't lie to us! You will note however, that I haven't trusted the rapidly developed app enough to let it near my phone ... :-D
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony "Common sense is so rare these days, it should be classified as a super power" - Random T-shirt AntiTwitter: @DalekDave is now a follower!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt -
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
The "funny" thing is that finding fifteen thousand plus new corona cases reduces mortality (from covid-19) in the UK. That is good, isn't it? At least that is the way some people see it, not considering mortality by the number of people who die, or the how large percentage of the population that dies, but by the percentage of people who are registered as being infected that dies. For a given death count, the more cases of infection that you register, the lower the percentage of people dying from it. Note that a country that doesn't count cases at all, zero reported cases, would have an infinite mortality rate even with a single corona death.
-
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
I know I've been on this soapbox before, but until the people in power realise that software development is not a science, these things will happen! They also need to realise that the vast majority of people involved in software development get an adrenalin rush whenever they are given an opportunity to work on shiny new stuff - and are nearly always hugely optimistic about the effort required and timescales. I'm not blaming the software guys. That would be like 'blaming' your Jack Russell Terrier for enthusiastically fetching the ball back, every time you throw it! ;) The people at the top are the ones that should be taking the hit on this. Firing a few politicians and senior civil servants, may not completely solve the problem - but it would certainly improve the mood of the nation. :laugh:
-
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
Having worked with NHS data in the past I wasn't surprised at the problem. There are going to be other problems that have been kept quiet. There is a dataset that contains hospital data (HES), however access is generally not available until some time after data was recorded in the SUS system from which HES is pulled. It can get a little complicated as HES is updated from SUS and while HES data is updated at periodic points from the SUS data, the SUS data can be updated after the HES update has taken place -> How we collect and process Hospital Episode Statistics (HES) data - NHS Digital[^] That's just hospital data, when it comes to GP(general practice) there are in the region of 15(I think) or so different systems used to collect data - there are companies who aggregate this data but again it can be a bit messy.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
-
But ... but ... it's a World-Beating System! The government says so, and it wouldn't lie to us! You will note however, that I haven't trusted the rapidly developed app enough to let it near my phone ... :-D
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony "Common sense is so rare these days, it should be classified as a super power" - Random T-shirt AntiTwitter: @DalekDave is now a follower!
They could just share the Excel file with everyone and ask us to add our name to it if we get sick. Our government is so incompetent I'm actually quite suprised they didn't do this.
-
They could just share the Excel file with everyone and ask us to add our name to it if we get sick. Our government is so incompetent I'm actually quite suprised they didn't do this.
You mean you didn't get your copy? :omg:
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony "Common sense is so rare these days, it should be classified as a super power" - Random T-shirt AntiTwitter: @DalekDave is now a follower!
-
I know I've been on this soapbox before, but until the people in power realise that software development is not a science, these things will happen! They also need to realise that the vast majority of people involved in software development get an adrenalin rush whenever they are given an opportunity to work on shiny new stuff - and are nearly always hugely optimistic about the effort required and timescales. I'm not blaming the software guys. That would be like 'blaming' your Jack Russell Terrier for enthusiastically fetching the ball back, every time you throw it! ;) The people at the top are the ones that should be taking the hit on this. Firing a few politicians and senior civil servants, may not completely solve the problem - but it would certainly improve the mood of the nation. :laugh:
5teveH wrote:
Firing at a few politicians and senior civil servants, may not completely solve the problem - but it would certainly improve the mood of the nation. :laugh:
FTFY!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony "Common sense is so rare these days, it should be classified as a super power" - Random T-shirt AntiTwitter: @DalekDave is now a follower!
-
5teveH wrote:
Firing at a few politicians and senior civil servants, may not completely solve the problem - but it would certainly improve the mood of the nation. :laugh:
FTFY!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony "Common sense is so rare these days, it should be classified as a super power" - Random T-shirt AntiTwitter: @DalekDave is now a follower!
-
(Note: disclaimer is at the end) There was a blip in reported covid cases in the UK as the system used to report the data to the government systems didn't work for a day so the results were rolled into the next batch of reported data. The cause of the blip was that the people collecting the results sent the data to the government systems using XLS files and due to the increasing number of cases the data exceeded the capacity for XLS files so they had to rejig the system to batch the data instead of putting it all in one file. As software engineers we can all agree there are issues here but as we're not privy to the inners of the system we can only guess at why certain decisions were made. If I was sending raw data to a third party system with (presumably) no API then I'd choose CSV files. However if the data was intended for human consumption then Excel is probably a decent solution. Rather than just raw numbers in cells it can have descriptive meta info at the top such as the source of the data, date it was taken on and so on, then the data itself in rows below. Another reason they might use XLS files is that it's easier to handle quotes, commas and other funny data, letting the proprietary format handle those issues, and while there are existing CSV frameworks that support that data off the shelf you have to bear in mind the time pressures to get this system working and the potential red tape etc in using a project written by some random that they found on github in a government system. Regardless of the reasons XLS files were used there were obvious failings in that no-one thought to test the capacity of the system so too many assumptions and no thinking about the long-game. So yeah there are flaws here obviously. What is annoying me though is the way this has been reported and reacted to on places like twitter, where you normally find nothing but well-informed, intellectually honest, non-partisan debate. 99.99% of posts I have seen about this have been of the same ilk, obviously written by people who don't understand the tech. The prevailing narrative being pushed is that the data is stored in Excel files rather than a database. However if that was the case then the data that was missed on the fateful day would probably be lost forever. The fact that it was produced later on in multiple files rather than one demonstrates that the data is indeed stored elsewhere and simply packed in XLS files for distribution to interested parties. No developer, no matter how poor, would really suggest to
I read somewhere that the Excel data was stored columnwise rather than rowwise so they hit the column limit of 16,384 instead of the row limit of 1,048,576. But the real problem is everyone blaming a "computer glitch", rather than explaining that the people writing the code misread a QA answer. :laugh:
-
They could just share the Excel file with everyone and ask us to add our name to it if we get sick. Our government is so incompetent I'm actually quite suprised they didn't do this.
-
I read somewhere that the Excel data was stored columnwise rather than rowwise so they hit the column limit of 16,384 instead of the row limit of 1,048,576. But the real problem is everyone blaming a "computer glitch", rather than explaining that the people writing the code misread a QA answer. :laugh:
Richard MacCutchan wrote:
But the real problem is everyone blaming a "computer glitch", rather than explaining that the people writing the code misread a QA answer. :laugh:
I wish that was a joke ... :sigh:
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony "Common sense is so rare these days, it should be classified as a super power" - Random T-shirt AntiTwitter: @DalekDave is now a follower!
-
The "funny" thing is that finding fifteen thousand plus new corona cases reduces mortality (from covid-19) in the UK. That is good, isn't it? At least that is the way some people see it, not considering mortality by the number of people who die, or the how large percentage of the population that dies, but by the percentage of people who are registered as being infected that dies. For a given death count, the more cases of infection that you register, the lower the percentage of people dying from it. Note that a country that doesn't count cases at all, zero reported cases, would have an infinite mortality rate even with a single corona death.
-
Which is exactly why there was hysteria in the UK early on; because testing was for a while effectively only being done on the very sick, the mortality rate was vastly above other countries.
Exactly. Yet... Even looking at deaths/1M population UK is quite high. Not quite as high as six countries on the other side of the pond, yet twelve times as high as Norway, ten times that of Finland. So the UK is hit quite hard by the Corona virus, even though not as hard as "registered cases / deaths" seems to suggest.
-
I read somewhere that the Excel data was stored columnwise rather than rowwise so they hit the column limit of 16,384 instead of the row limit of 1,048,576. But the real problem is everyone blaming a "computer glitch", rather than explaining that the people writing the code misread a QA answer. :laugh:
-
Exactly. Yet... Even looking at deaths/1M population UK is quite high. Not quite as high as six countries on the other side of the pond, yet twelve times as high as Norway, ten times that of Finland. So the UK is hit quite hard by the Corona virus, even though not as hard as "registered cases / deaths" seems to suggest.
-
5teveH wrote:
Firing at a few politicians and senior civil servants, may not completely solve the problem - but it would certainly improve the mood of the nation. :laugh:
FTFY!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony "Common sense is so rare these days, it should be classified as a super power" - Random T-shirt AntiTwitter: @DalekDave is now a follower!
Might depend on how well you aim.
Wrong is evil and must be defeated. - Jeff Ello Never stop dreaming - Freddie Kruger
-
I read somewhere that the Excel data was stored columnwise rather than rowwise so they hit the column limit of 16,384 instead of the row limit of 1,048,576. But the real problem is everyone blaming a "computer glitch", rather than explaining that the people writing the code misread a QA answer. :laugh:
Richard MacCutchan wrote:
I read somewhere that the Excel data was stored columnwise rather than rowwise
I also read that and considered including it in the OP. I think it's one of the clearer bits of misinformation, I mean who would seriously propose that as a solution? Even the worst of the worst in QA would suggest a new sheet per case rather than a new column.
