Structural Testing

[Marc Richarme]

Disclaimer: This post might be considered a programming question to some people. http://www.imm.dtu.dk/~ah/Undervisning/02100+02199/E02/files/sestofttest.pdf[^] Is anyone actually doing such structural tests in the real world? If yes, how much time does it take?

Cheers,
Marc

:beer: Click to see my *real* signature :beer:

[Sean Winstead]

Structural tests sound similar to what I know as unit tests. I've written unit tests for work & personal projects. My gut feeling is that it doubles the development time. But I write tests as I finish each class, instead of waiting until the end. The wonderful thing about doing it this way is that a) I know I'm building new code on a tested, solid foundation and b) I immediately see situations where I have broken working code with new changes. SO my guess is that it takes less time than writing the tests at the end. As far as coverage goes, I've been able to reach a maximum of 85% code coverage (measured with a coverage analysis tool). There's a certain percentage point where it is cost-prohibitive to continue writing tests in order to achieve 100% coverage. And even at 100% coverage, you're probably missing some scenario that could lead to a failure. Sean Winstead

[Nitron]

Sean Winstead wrote: Structural tests sound similar to what I know as unit tests. Same here. Sean Winstead wrote: My gut feeling is that it doubles the development time. I highly disagree! Our unit test is merely our first phase of testing. If problems can be caught at unit test, it is much cheaper to fix problems there than finding the problems in the lab during system-level and failure mode testing! Unit test can be a great time saver IMO. Especially if your software is to meet SEI Level 4[^] or higher standards. - Nitron

---

"Those that say a task is impossible shouldn't interrupt the ones who are doing it." - Chinese Proverb

[Giles]

Nice idea, but somehow I don't think its practical to test in this way the whole app, for the kind of projects I do on the whole anyway. I would consider doing it for important parts of the code though, and I could see how this would be useful for someone writting embeded software, where failure is not a option e.g. auto pilot in a comercial airliner.

[Roger Wright]

This sounds like the software equivalent of in-circuit hardware test, though the module test approach Sean mentions is certainly the best way to approach it. My own experience is in hardware test; I've only managed software testing, not done it myself in a formal fashiion. But from a cost perspective, the two are very similar. Structural test of software exercises each function, especially control structures in a program, to ensure that they work as intended. In-circuit hardware testing checks each component in an assembly, to ensure that each is the right value, is installed correctly, and behaves according to its spec. In terms of equipment cost and design labor, in-circuit testing costs 10% to 20% of what a full functional test does. It also provides a wealth of information for fault prediction and diagnosis, which functional testing does not. What it does not tell you is whether the design is any good. All the parts can work perfectly, but the component still fail in service, if the design is fundamentally flawed - that really pisses off the product designers, but life's tough. Structural testing tests what is there, not what should have been included; if a possible event has not been handled, it won't find that. Module testing at each level of integration is an overall time saver, even though the product designer often feels that it is a waste of his time to do it. In hardware design and test, the cost of finding and fixing a problem increases roughly ten-fold at each level of integration. If a fault in a hybrid circuit costs $1 to find and fix, once it's added to a circuit board, the cost is $10, and after it goes into a chassis (or,in my experience, a missile guidance section) it rises to $100. The same is true for software - if it takes an hour to test and correct a faulty function, it will take about 10 hours to find the fault once it's added to a project. And that cost will again escalate by a factor of about 10 when the project is added to other projects that make up the next level of integration in the product. Hardware or software, structural testing adds value, especially if fault prediction and diagnosis is important, and it does so at a much lower cost over the life cycle of the product. Functional test is still required to ensure that the designer really understood the requirements, though, but it needn't be overly complex as diagnostic functional tests often are. Module testing will always reduce costs when viewed from the life cycle perspective, though they will certainly i

[Sean Winstead]

I highly disagree! Our unit test is merely our first phase of testing. If problems can be caught at unit test, it is much cheaper to fix problems there than finding the problems in the lab during system-level and failure mode testing! Unit test can be a great time saver IMO. Especially if your software is to meet SEI Level 4[^] or higher standards. You're reading too much into my reply. Unit testing does extend my development time because I am writing unit tests and fixing the bugs/design problems they so well point out. Like you, the value in writing unit tests far outweighs the cost that comes into play if the unit tests are not written. And if it doubles my development time then it doubles my development time. Sean Winstead

[mbh ito]

I've used automated module and integration testing for some time now, and my experience is that it both cuts initial development time, testing AND subsequent making of the next versions of the product. Granted there is some time spent writing the testcases, but this time saved by spending less time looking for bugs already during intial development. We do a lot of refactoring of our code (constant improvement ?!?), and run the testsuite several times a day. In doing this we catch newly introduced errors close to the point in time they have been introduced, and they are therefore a lot easier to locate and squash. Of course the tests do not catch everything, and when discovering a un-detected bug, we write a testcase which catch the error befor efixing the bug. This way the testsuite constantly get better and better. There is no way that I would go back to developing without an automated testing system. PERIOD!