Defect ratios [modified]
-
Defects are good. When I was with the State of Maine I told the Director to expect about 500 defects with the product when delivered. He was shocked. When the product was delivered and he started to see what a defect was, how they were fixed, and their importance he quickly became very happy. The bottom line is defects with Agile help deliver a quality application that MATCHES the client expectation. That said however, you mention your product is in Acceptance testing which means there should not be any defects, period. Acceptance comes after delivery. Your project should have had multiple beta deliveries planned so you could have identified and corrected those issues and shown a progressive track record of improving quality. Right now you are trying to deliver a product that is known to be defective. When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. A man said to the universe: "Sir I exist!" "However," replied the universe, "The fact has not created in me A sense of obligation." --Stephen Crane
Ennis Ray Lynch, Jr. wrote:
ou mention your product is in Acceptance testing
No, I said they are refusing to even begin it. Whether the module has any reported defects or not.
Ennis Ray Lynch, Jr. wrote:
Your project should have had multiple beta deliveries planned
That's the point. It has, the customer hasn't done any of the tests though.
Ennis Ray Lynch, Jr. wrote:
When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Really not a good comparison IMO. Bad concrete causes structural failure that can cost lives. A software defect may only cause an incorrect total to be calculated, hardly on the same scale.
I know the language. I've read a book. - _Madmatt
-
EliottA wrote:
4MIL LOC, and reportedly close to 20k defects, roughly 0.005% defects
Isn't that 0.5% ?
Maybe, I might be extremely tired as it's the end of the day and close to a vacation. Right now, all inquiries are being placed in the 'I don't care to look at, address or resolve this until 2010' bin. Please place your inquiry there and I promise I will [never] get to it. Thank you.
Check out the CodeProject forum Guidelines[^] The original soapbox 1.0 is back![^]
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
Mark Nischalke wrote:
9000 LOC and 52 reported defects during initial testing. 173:1 or 0.02%
173:1 is 0.57%, not 0.02%
Mark Nischalke wrote:
the client has refused to start testing
Not surprised. That ratio seems very high to me. Scaling up proportionally (and yes, I realise that's not a "scientifically valid" thing to do), our main product, that has about 400K LOC, would have over 2,300 defects. We wouldn't release anything outside of the development team with that defect level, and would never put it in front of a paying client.
-
Ennis Ray Lynch, Jr. wrote:
ou mention your product is in Acceptance testing
No, I said they are refusing to even begin it. Whether the module has any reported defects or not.
Ennis Ray Lynch, Jr. wrote:
Your project should have had multiple beta deliveries planned
That's the point. It has, the customer hasn't done any of the tests though.
Ennis Ray Lynch, Jr. wrote:
When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Really not a good comparison IMO. Bad concrete causes structural failure that can cost lives. A software defect may only cause an incorrect total to be calculated, hardly on the same scale.
I know the language. I've read a book. - _Madmatt
Software can be life or death[^]
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. A man said to the universe: "Sir I exist!" "However," replied the universe, "The fact has not created in me A sense of obligation." --Stephen Crane
-
Mark Nischalke wrote:
9000 LOC and 52 reported defects during initial testing. 173:1 or 0.02%
173:1 is 0.57%, not 0.02%
Mark Nischalke wrote:
the client has refused to start testing
Not surprised. That ratio seems very high to me. Scaling up proportionally (and yes, I realise that's not a "scientifically valid" thing to do), our main product, that has about 400K LOC, would have over 2,300 defects. We wouldn't release anything outside of the development team with that defect level, and would never put it in front of a paying client.
Perhaps I'm having a senior moment but, 173/9000 = 0.019 Actually I just realized it should have been 52/9000, so it's 0.005. Guess I was having a senior moment :( Perhaps I wasn't clear enough. This product is not released, only the first round of development is complete.
I know the language. I've read a book. - _Madmatt
-
Perhaps I'm having a senior moment but, 173/9000 = 0.019 Actually I just realized it should have been 52/9000, so it's 0.005. Guess I was having a senior moment :( Perhaps I wasn't clear enough. This product is not released, only the first round of development is complete.
I know the language. I've read a book. - _Madmatt
Mark Nischalke wrote:
Perhaps I'm having a senior moment
You are. 52 / 9000 * 100 = 0.577
Mark Nischalke wrote:
This product is not released, only the first round of development is complete.
Doesn't matter. With that defect level, I don't think it should be put in front of a paying client.
-
Mark Nischalke wrote:
Perhaps I'm having a senior moment
You are. 52 / 9000 * 100 = 0.577
Mark Nischalke wrote:
This product is not released, only the first round of development is complete.
Doesn't matter. With that defect level, I don't think it should be put in front of a paying client.
You're right I forgot to complete the calculation. Long day
I know the language. I've read a book. - _Madmatt
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
Why is it that we have to suffer under the aegis of "defects" et al, while no one seems to track various ratios like: defects to managers defects to requirement changes defects to equipment quality defects to number of times we hear "that utility / developer tool / etc. is too expensive" defects to bonuses (hmmm, I can predict a divide by zero error here) defects to number of vacation days defects to number of hours worked each week above 40 defects to work schedule flexibility Ah, I could go on, but if I do, I might defect to another profession. ;) Marc
I'm not overthinking the problem, I just felt like I needed a small, unimportant, uninteresting rant! - Martin Hart Turner
-
Why is it that we have to suffer under the aegis of "defects" et al, while no one seems to track various ratios like: defects to managers defects to requirement changes defects to equipment quality defects to number of times we hear "that utility / developer tool / etc. is too expensive" defects to bonuses (hmmm, I can predict a divide by zero error here) defects to number of vacation days defects to number of hours worked each week above 40 defects to work schedule flexibility Ah, I could go on, but if I do, I might defect to another profession. ;) Marc
I'm not overthinking the problem, I just felt like I needed a small, unimportant, uninteresting rant! - Martin Hart Turner
Marc Clifton wrote:
defects to number of vacation days defects to number of hours worked each week above 40
For a time I actually kept a metric of defects & productivity vs hours per day and defects & productivity vs hours of sleep. I found, for myself at least, that sleeping less than 6.5 hours hammered my productivity by 40% and increased the number of mistakes/errors/defects by over 55%. Also, when I spent more than 8 hours a day cutting code for more than three days in a row my productivity took about a 10% hit but the mistakes/errors/defect went up by roughly 21%. Pretty wild.
And above all things, never think that you're not good enough yourself. A man should never think that. My belief is that in life people will take you at your own reckoning. --Isaac Asimov Avoid the crowd. Do your own thinking independently. Be the chess player, not the chess piece. --Ralph Charell
-
Why is it that we have to suffer under the aegis of "defects" et al, while no one seems to track various ratios like: defects to managers defects to requirement changes defects to equipment quality defects to number of times we hear "that utility / developer tool / etc. is too expensive" defects to bonuses (hmmm, I can predict a divide by zero error here) defects to number of vacation days defects to number of hours worked each week above 40 defects to work schedule flexibility Ah, I could go on, but if I do, I might defect to another profession. ;) Marc
I'm not overthinking the problem, I just felt like I needed a small, unimportant, uninteresting rant! - Martin Hart Turner
Marc Clifton wrote:
defects to bonuses (hmmm, I can predict a divide by zero error here)
LOL. defects to old hardware defects to free beer (probably some kind of asymtopic relationship here, I think) :cool:
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra]
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
What it depends on most, is your resources. How much time were you given for in depth testing, as opposed to basic testing of features as you go ? Is there a proper testing team ? Are there unit tests, and if there were, would they have picked up these issues ? In my experience, an initial iteration is likely to have a decent number of issues, when first tested. Most of those issues are likely to be things that can be fixed very quickly. I would typically expect to put in 1-2 days and knock a list like that down to a fraction of how it started, and then perhaps spend a day or two fixing the most major of the remaining issues. I would sign a contract typically that expected 1/3 up front, 1/3 on delivery of the initial version ( which I'd consider to be the version you'd turn in AFTER doing what I just said to your 52 bugs ), and 1/3 on acceptance of a final version. If you're expecting full payment before the bugs have been fixed, I think the client has a point, if you're talking about an iteration, then I think you have a point.
Christian Graus Driven to the arms of OSX by Vista. Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
Mark Nischalke wrote:
As a total it makes 57%
Nope. It's 0.57%, which is still awful, but a decent level for beginning in-house testing. I wouldn't want a client to see it, though, or my boss. :-D I realize that Microsoft has institutionalized the concept of selling defective items, but that's no excuse for buying into their diseased imaginings. That's good marketing, not engineering. Alpha and beta deliverables have defects, mockups and prototypes have defects. Release Candidates might have a few, if they're really complex. Products do not.
"A Journey of a Thousand Rest Stops Begins with a Single Movement"
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
I would consider 52 errors in 9 kloc to be rather high.
-- Kein Mitleid Für Die Mehrheit
-
Why is it that we have to suffer under the aegis of "defects" et al, while no one seems to track various ratios like: defects to managers defects to requirement changes defects to equipment quality defects to number of times we hear "that utility / developer tool / etc. is too expensive" defects to bonuses (hmmm, I can predict a divide by zero error here) defects to number of vacation days defects to number of hours worked each week above 40 defects to work schedule flexibility Ah, I could go on, but if I do, I might defect to another profession. ;) Marc
I'm not overthinking the problem, I just felt like I needed a small, unimportant, uninteresting rant! - Martin Hart Turner
You know, you should do some research! I would love to know read a paper/dissertation that covers this line of reasoning. :)
-- Kein Mitleid Für Die Mehrheit
-
I would consider 52 errors in 9 kloc to be rather high.
-- Kein Mitleid Für Die Mehrheit
Yes, I was looking at it incorrectly and not explaining well, fustration and a long day. Of the 52 reported at the end of development, only 10 remain after a week of cleanup and all but 2 affect auxilary apps, like report formatting, not the actual app.
I know the language. I've read a book. - _Madmatt
-
What it depends on most, is your resources. How much time were you given for in depth testing, as opposed to basic testing of features as you go ? Is there a proper testing team ? Are there unit tests, and if there were, would they have picked up these issues ? In my experience, an initial iteration is likely to have a decent number of issues, when first tested. Most of those issues are likely to be things that can be fixed very quickly. I would typically expect to put in 1-2 days and knock a list like that down to a fraction of how it started, and then perhaps spend a day or two fixing the most major of the remaining issues. I would sign a contract typically that expected 1/3 up front, 1/3 on delivery of the initial version ( which I'd consider to be the version you'd turn in AFTER doing what I just said to your 52 bugs ), and 1/3 on acceptance of a final version. If you're expecting full payment before the bugs have been fixed, I think the client has a point, if you're talking about an iteration, then I think you have a point.
Christian Graus Driven to the arms of OSX by Vista. Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
Of the 52 reported at the end of development all but 10 were fixed within a week. 8 of the remaing were things like report formatting, low priority
I know the language. I've read a book. - _Madmatt
-
Of the 52 reported at the end of development all but 10 were fixed within a week. 8 of the remaing were things like report formatting, low priority
I know the language. I've read a book. - _Madmatt
OK, well, now that you clarify that, I tend to agree with you. There's always going to be issues at the start of testing. Of course, the caveat remains, if the client is only required to pay on completion, then I can see why they may feel they need to keep some leverage until the bugs are all fixed. That's why I prefer to be paid in 3 installments, so that I'm not held to ransom for all the money based on one or two niggly issues.
Christian Graus Driven to the arms of OSX by Vista. Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
-
Ennis Ray Lynch, Jr. wrote:
ou mention your product is in Acceptance testing
No, I said they are refusing to even begin it. Whether the module has any reported defects or not.
Ennis Ray Lynch, Jr. wrote:
Your project should have had multiple beta deliveries planned
That's the point. It has, the customer hasn't done any of the tests though.
Ennis Ray Lynch, Jr. wrote:
When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Really not a good comparison IMO. Bad concrete causes structural failure that can cost lives. A software defect may only cause an incorrect total to be calculated, hardly on the same scale.
I know the language. I've read a book. - _Madmatt
Mark Nischalke wrote:
Really not a good comparison IMO. Bad concrete causes structural failure that can cost lives. A software defect may only cause an incorrect total to be calculated, hardly on the same scale.
I hope you don't work for NASA. Even if your product only produces an incorrect total on a report if this figure is used to make a safety decision that may come back to bite someone in the future it is a big deal. Obviously, I don't know from what you have said what it is your system does. If it orders 500 items instead of 5000 items then it depends on the item. If the items are containers of milk it is an inconvenience. If it certifies that 500 fire extinguishers instead of 5000 are enough for a cruise ship to pass certification than it is a big deal.
Simply Elegant Designs JimmyRopes Designs
Think inside the box! ProActive Secure Systems
I'm on-line therefore I am. JimmyRopes -
Mark Nischalke wrote:
As a total it makes 57%
Nope. It's 0.57%, which is still awful, but a decent level for beginning in-house testing. I wouldn't want a client to see it, though, or my boss. :-D I realize that Microsoft has institutionalized the concept of selling defective items, but that's no excuse for buying into their diseased imaginings. That's good marketing, not engineering. Alpha and beta deliverables have defects, mockups and prototypes have defects. Release Candidates might have a few, if they're really complex. Products do not.
"A Journey of a Thousand Rest Stops Begins with a Single Movement"
Roger Wright wrote:
I realize that Microsoft has institutionalized the concept of selling defective items
:laugh: It is strange how we have slipped into that mentality (Windows 95 release long before it was ready) when for such a long time a quality product was essential to a successful business.
Simply Elegant Designs JimmyRopes Designs
Think inside the box! ProActive Secure Systems
I'm on-line therefore I am. JimmyRopes