Defect ratios [modified]
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
Here are the typical figures , (From Estimating Software Costs[^])
LOC (in thousands)
Errors/1000 LOC
< 2
0-25
2-16
0-40
16-64
0.5-50
64-512
2-70
>512
4-100
It is no brainer that, higher the LOC more the error ratio.
-
Hmm... Our metric is simply the number of bugs reported. It's not pro-rated with the number of lines of code. That said, our active bug list usually averages less than 100 bugs per product, and our products average 500,000 LOC. 52 defects in 9,000 lines of code does seem rather high. Do you classify your bugs? We typically have 'critical' (the world is coming to an end, i.e. the customer ain't paying), 'major' (someone's panties are in a bunch), 'minor' (Q.A. is pissed), and 'cosmetic' (somebody prefers pink). If you had say, 1 critical, 3 major, 20 minor, and 28 cosmetic, that's not bad. If it's more like 18 critical, 20 major, 4 minor, and 10 cosmetic, then you've got troubles.
Mark Nischalke wrote:
the client has refused to start acceptance testing
That obviously begs the question: if they aren't testing, then what are they basing their refusal on? An acceptance test either passes or fails. In either case, there are certain contractual obligations for both you and the client. If it passes, the client is supposed to complete payment on the contract. If it fails, you go back and fix things until it passes, or you and the client modify the terms of the acceptance test.
Software Zen:
delete this;Yes, breaking them down there are only two critical defects. The contractual aspects are of course beyond me. I wasn't privy to the details since I'm only doing the work, but this company is very immature and I'm sure didn't consider these things when signing the agreement with the client.
I know the language. I've read a book. - _Madmatt
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
I've heard and read of people using this as a metric but it seems to me to be too big of a 'bucket' to really be meaningful. When I look at code quality, I tend to categorize defects based on severity and I dump all defects into a pareto chart based on the cause as well.
And above all things, never think that you're not good enough yourself. A man should never think that. My belief is that in life people will take you at your own reckoning. --Isaac Asimov Avoid the crowd. Do your own thinking independently. Be the chess player, not the chess piece. --Ralph Charell
-
Hmm... Our metric is simply the number of bugs reported. It's not pro-rated with the number of lines of code. That said, our active bug list usually averages less than 100 bugs per product, and our products average 500,000 LOC. 52 defects in 9,000 lines of code does seem rather high. Do you classify your bugs? We typically have 'critical' (the world is coming to an end, i.e. the customer ain't paying), 'major' (someone's panties are in a bunch), 'minor' (Q.A. is pissed), and 'cosmetic' (somebody prefers pink). If you had say, 1 critical, 3 major, 20 minor, and 28 cosmetic, that's not bad. If it's more like 18 critical, 20 major, 4 minor, and 10 cosmetic, then you've got troubles.
Mark Nischalke wrote:
the client has refused to start acceptance testing
That obviously begs the question: if they aren't testing, then what are they basing their refusal on? An acceptance test either passes or fails. In either case, there are certain contractual obligations for both you and the client. If it passes, the client is supposed to complete payment on the contract. If it fails, you go back and fix things until it passes, or you and the client modify the terms of the acceptance test.
Software Zen:
delete this;Gary Wheeler wrote:
52 defects in 9,000 lines of code does seem rather high.
I would say this is high. In a typical 75000 to 90000 line program that I write in a 12 to 18 months there better not be 52 detectable bugs when it hits production.
Gary Wheeler wrote:
Do you classify your bugs? We typically have 'critical' (the world is coming to an end, i.e. the customer ain't paying), 'major' (someone's panties are in a bunch), 'minor' (Q.A. is pissed), and 'cosmetic' (somebody prefers pink). If you had say, 1 critical, 3 major, 20 minor, and 28 cosmetic, that's not bad. If it's more like 18 critical, 20 major, 4 minor, and 10 cosmetic, then you've got troubles.
Very important what defects are classified as. On my average projects (ones where I am the only programmer) 0 or 1 critcal, 0 or 1 major, 0 to 3 minor and 2 to 10 cosmetic at the time it goes into production. Some of the minor and cosmetic are known at the time of production just that there was not enough time to fix or fixing them could break the design.
John
-
Gary Wheeler wrote:
52 defects in 9,000 lines of code does seem rather high.
I would say this is high. In a typical 75000 to 90000 line program that I write in a 12 to 18 months there better not be 52 detectable bugs when it hits production.
Gary Wheeler wrote:
Do you classify your bugs? We typically have 'critical' (the world is coming to an end, i.e. the customer ain't paying), 'major' (someone's panties are in a bunch), 'minor' (Q.A. is pissed), and 'cosmetic' (somebody prefers pink). If you had say, 1 critical, 3 major, 20 minor, and 28 cosmetic, that's not bad. If it's more like 18 critical, 20 major, 4 minor, and 10 cosmetic, then you've got troubles.
Very important what defects are classified as. On my average projects (ones where I am the only programmer) 0 or 1 critcal, 0 or 1 major, 0 to 3 minor and 2 to 10 cosmetic at the time it goes into production. Some of the minor and cosmetic are known at the time of production just that there was not enough time to fix or fixing them could break the design.
John
I should have clarified that 52 is after the first round of testing, not release. They were are also classified with about 30% being ones due to things like incorrect requirements.
I know the language. I've read a book. - _Madmatt
-
I should have clarified that 52 is after the first round of testing, not release. They were are also classified with about 30% being ones due to things like incorrect requirements.
I know the language. I've read a book. - _Madmatt
That makes a big difference as well. And incorrect requirements should not be the programmers fault. Although at times you do get blamed for not being able to guess what the actual required functionally is..
John
-
That makes a big difference as well. And incorrect requirements should not be the programmers fault. Although at times you do get blamed for not being able to guess what the actual required functionally is..
John
CIS 406: Reading the Mind of the Customer They offered that course after I graduated :laugh:
I know the language. I've read a book. - _Madmatt
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
Defects are good. When I was with the State of Maine I told the Director to expect about 500 defects with the product when delivered. He was shocked. When the product was delivered and he started to see what a defect was, how they were fixed, and their importance he quickly became very happy. The bottom line is defects with Agile help deliver a quality application that MATCHES the client expectation. That said however, you mention your product is in Acceptance testing which means there should not be any defects, period. Acceptance comes after delivery. Your project should have had multiple beta deliveries planned so you could have identified and corrected those issues and shown a progressive track record of improving quality. Right now you are trying to deliver a product that is known to be defective. When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. A man said to the universe: "Sir I exist!" "However," replied the universe, "The fact has not created in me A sense of obligation." --Stephen Crane
-
Depends on the size. We have an application that is close to 4MIL LOC, and reportedly close to 20k defects, roughly 0.005% defects. A lot of it is generated code, so I guess you could factor that out, and the DAL to bring it down to about 1MIL bringing it closer to your 0.02%, which is acceptable enough for names like Puma, so it should be acceptable enough for your client. But in your case, I'd say the magic number is the number that makes the comfortable enough to pay you :)
Check out the CodeProject forum Guidelines[^] The original soapbox 1.0 is back![^]
EliottA wrote:
4MIL LOC, and reportedly close to 20k defects, roughly 0.005% defects
Isn't that 0.5% ?
-
Defects are good. When I was with the State of Maine I told the Director to expect about 500 defects with the product when delivered. He was shocked. When the product was delivered and he started to see what a defect was, how they were fixed, and their importance he quickly became very happy. The bottom line is defects with Agile help deliver a quality application that MATCHES the client expectation. That said however, you mention your product is in Acceptance testing which means there should not be any defects, period. Acceptance comes after delivery. Your project should have had multiple beta deliveries planned so you could have identified and corrected those issues and shown a progressive track record of improving quality. Right now you are trying to deliver a product that is known to be defective. When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. A man said to the universe: "Sir I exist!" "However," replied the universe, "The fact has not created in me A sense of obligation." --Stephen Crane
Ennis Ray Lynch, Jr. wrote:
ou mention your product is in Acceptance testing
No, I said they are refusing to even begin it. Whether the module has any reported defects or not.
Ennis Ray Lynch, Jr. wrote:
Your project should have had multiple beta deliveries planned
That's the point. It has, the customer hasn't done any of the tests though.
Ennis Ray Lynch, Jr. wrote:
When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Really not a good comparison IMO. Bad concrete causes structural failure that can cost lives. A software defect may only cause an incorrect total to be calculated, hardly on the same scale.
I know the language. I've read a book. - _Madmatt
-
EliottA wrote:
4MIL LOC, and reportedly close to 20k defects, roughly 0.005% defects
Isn't that 0.5% ?
Maybe, I might be extremely tired as it's the end of the day and close to a vacation. Right now, all inquiries are being placed in the 'I don't care to look at, address or resolve this until 2010' bin. Please place your inquiry there and I promise I will [never] get to it. Thank you.
Check out the CodeProject forum Guidelines[^] The original soapbox 1.0 is back![^]
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
Mark Nischalke wrote:
9000 LOC and 52 reported defects during initial testing. 173:1 or 0.02%
173:1 is 0.57%, not 0.02%
Mark Nischalke wrote:
the client has refused to start testing
Not surprised. That ratio seems very high to me. Scaling up proportionally (and yes, I realise that's not a "scientifically valid" thing to do), our main product, that has about 400K LOC, would have over 2,300 defects. We wouldn't release anything outside of the development team with that defect level, and would never put it in front of a paying client.
-
Ennis Ray Lynch, Jr. wrote:
ou mention your product is in Acceptance testing
No, I said they are refusing to even begin it. Whether the module has any reported defects or not.
Ennis Ray Lynch, Jr. wrote:
Your project should have had multiple beta deliveries planned
That's the point. It has, the customer hasn't done any of the tests though.
Ennis Ray Lynch, Jr. wrote:
When I worked in the concrete industry shipping defective concrete was a no-no (if you get caught in QA you have to pay to redo the entire pour)
Really not a good comparison IMO. Bad concrete causes structural failure that can cost lives. A software defect may only cause an incorrect total to be calculated, hardly on the same scale.
I know the language. I've read a book. - _Madmatt
Software can be life or death[^]
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting. A man said to the universe: "Sir I exist!" "However," replied the universe, "The fact has not created in me A sense of obligation." --Stephen Crane
-
Mark Nischalke wrote:
9000 LOC and 52 reported defects during initial testing. 173:1 or 0.02%
173:1 is 0.57%, not 0.02%
Mark Nischalke wrote:
the client has refused to start testing
Not surprised. That ratio seems very high to me. Scaling up proportionally (and yes, I realise that's not a "scientifically valid" thing to do), our main product, that has about 400K LOC, would have over 2,300 defects. We wouldn't release anything outside of the development team with that defect level, and would never put it in front of a paying client.
Perhaps I'm having a senior moment but, 173/9000 = 0.019 Actually I just realized it should have been 52/9000, so it's 0.005. Guess I was having a senior moment :( Perhaps I wasn't clear enough. This product is not released, only the first round of development is complete.
I know the language. I've read a book. - _Madmatt
-
Perhaps I'm having a senior moment but, 173/9000 = 0.019 Actually I just realized it should have been 52/9000, so it's 0.005. Guess I was having a senior moment :( Perhaps I wasn't clear enough. This product is not released, only the first round of development is complete.
I know the language. I've read a book. - _Madmatt
Mark Nischalke wrote:
Perhaps I'm having a senior moment
You are. 52 / 9000 * 100 = 0.577
Mark Nischalke wrote:
This product is not released, only the first round of development is complete.
Doesn't matter. With that defect level, I don't think it should be put in front of a paying client.
-
Mark Nischalke wrote:
Perhaps I'm having a senior moment
You are. 52 / 9000 * 100 = 0.577
Mark Nischalke wrote:
This product is not released, only the first round of development is complete.
Doesn't matter. With that defect level, I don't think it should be put in front of a paying client.
You're right I forgot to complete the calculation. Long day
I know the language. I've read a book. - _Madmatt
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
Why is it that we have to suffer under the aegis of "defects" et al, while no one seems to track various ratios like: defects to managers defects to requirement changes defects to equipment quality defects to number of times we hear "that utility / developer tool / etc. is too expensive" defects to bonuses (hmmm, I can predict a divide by zero error here) defects to number of vacation days defects to number of hours worked each week above 40 defects to work schedule flexibility Ah, I could go on, but if I do, I might defect to another profession. ;) Marc
I'm not overthinking the problem, I just felt like I needed a small, unimportant, uninteresting rant! - Martin Hart Turner
-
Why is it that we have to suffer under the aegis of "defects" et al, while no one seems to track various ratios like: defects to managers defects to requirement changes defects to equipment quality defects to number of times we hear "that utility / developer tool / etc. is too expensive" defects to bonuses (hmmm, I can predict a divide by zero error here) defects to number of vacation days defects to number of hours worked each week above 40 defects to work schedule flexibility Ah, I could go on, but if I do, I might defect to another profession. ;) Marc
I'm not overthinking the problem, I just felt like I needed a small, unimportant, uninteresting rant! - Martin Hart Turner
Marc Clifton wrote:
defects to number of vacation days defects to number of hours worked each week above 40
For a time I actually kept a metric of defects & productivity vs hours per day and defects & productivity vs hours of sleep. I found, for myself at least, that sleeping less than 6.5 hours hammered my productivity by 40% and increased the number of mistakes/errors/defects by over 55%. Also, when I spent more than 8 hours a day cutting code for more than three days in a row my productivity took about a 10% hit but the mistakes/errors/defect went up by roughly 21%. Pretty wild.
And above all things, never think that you're not good enough yourself. A man should never think that. My belief is that in life people will take you at your own reckoning. --Isaac Asimov Avoid the crowd. Do your own thinking independently. Be the chess player, not the chess piece. --Ralph Charell
-
Why is it that we have to suffer under the aegis of "defects" et al, while no one seems to track various ratios like: defects to managers defects to requirement changes defects to equipment quality defects to number of times we hear "that utility / developer tool / etc. is too expensive" defects to bonuses (hmmm, I can predict a divide by zero error here) defects to number of vacation days defects to number of hours worked each week above 40 defects to work schedule flexibility Ah, I could go on, but if I do, I might defect to another profession. ;) Marc
I'm not overthinking the problem, I just felt like I needed a small, unimportant, uninteresting rant! - Martin Hart Turner
Marc Clifton wrote:
defects to bonuses (hmmm, I can predict a divide by zero error here)
LOL. defects to old hardware defects to free beer (probably some kind of asymtopic relationship here, I think) :cool:
Chris Meech I am Canadian. [heard in a local bar] In theory there is no difference between theory and practice. In practice there is. [Yogi Berra]
-
I'm curious as to everyones view as to what is an acceptable, or expected, ratio of defects per lines of code. I know, it should be zero, but let's stick to reality. ;P I currently have a project with about 9000 LOC and 52 reported defects during initial testing. 173:1 or 0.006% I consider this good, but a manager is frustrated with the "high" number, the client has refused to start testing and most importantly, is withholding payment. [edit] I should have clarified that 52 is after the first round of testing, not release. They are also classified, high, medium, low, with about 30% being low due to things like incorrect or necessary changes to requirements and only two high priority. [edit] As Electron Shepherd pointed out my calculations were flawed (I need a vacation). As a total it makes 57% which is very bad, but factoring out the low priority missing requirement defects lowers it a good deal and considering only two high priority defects it really isn't bad at all.
I know the language. I've read a book. - _Madmatt
modified on Friday, December 18, 2009 2:49 PM
What it depends on most, is your resources. How much time were you given for in depth testing, as opposed to basic testing of features as you go ? Is there a proper testing team ? Are there unit tests, and if there were, would they have picked up these issues ? In my experience, an initial iteration is likely to have a decent number of issues, when first tested. Most of those issues are likely to be things that can be fixed very quickly. I would typically expect to put in 1-2 days and knock a list like that down to a fraction of how it started, and then perhaps spend a day or two fixing the most major of the remaining issues. I would sign a contract typically that expected 1/3 up front, 1/3 on delivery of the initial version ( which I'd consider to be the version you'd turn in AFTER doing what I just said to your 52 bugs ), and 1/3 on acceptance of a final version. If you're expecting full payment before the bugs have been fixed, I think the client has a point, if you're talking about an iteration, then I think you have a point.
Christian Graus Driven to the arms of OSX by Vista. Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.