Builds, bugs, testing, releases
-
My team is establishing an automated build environment. None of us has any real world experience working with such an environment. We're running into questions about once a build is created, how should test releases, testing, bug tracking, bug fixes, and production releases be "handled"? What I mean by handled is how the process in which these activities are performed and the data that is collected should all be organized. Some example questions: 1) Should each automated build be released for testing, or only selected builds? 2) Does a developer need to indicate in each build which bugs are fixed, or only in a release that is tested? ...and so on. I'm expecting the answers to these questions will all begin with "It depends...", but I'd like to find some resources that will help us make the best decision for how our team does work. Articles, books, or personal experience is welcomed.
I can only give you my experience. In the past I have done double duty as the build engineer. My current job is the first time this hasn't been the case. The specifics I mention are for my latest company, but reflect how I've also done it in the past. Item 1: All automated builds are available for testing. Testers usually grab the new build, though very rarely they may be in the middle of something and may skip a build. However, they are technically allowed to file bugs only against the latest build (this hasn't been a problem--writing coherent bug reports is a bigger issue.) Once we get to beta testing, all automated builds are available to both internal and external beta testers. We use RTPatch for delivering updates and all builds have RTP update files created for them from the previous build, which gives them the option of doing an update or downloading the entire install (which is quite big.) Item 2: The consensus I've found over the years is that bug fixes should identify the build number in which they were fixed. (The engineer would specific a date and the next build number for benefit of the testers who would then state in the report the earliest build in which they verified the fix.) This requires that versioning be done in a predictable manner. At my current job, we do a "major" build every Tuesday morning, where we increment the build number by 10. For interim builds (there were two this week) the build number is incremented by one. (We only specify the next major build number for fixes.) With every automated build, a notification is sent out containing all the major fixes and whether they are complete or only partially complete, and other readme type things (like; delete all data files and reimport your data.)(Note: To prevent testers from thinking us complete idiots keep the list short, we don't list all the fixes.) (Small confession; the automated build scripts were written years ago by a developer who not only loved perl, but loved to make his code as convoluted as possible and, to add insult to injury, was clueless about how Visual Studio, or even batch files for that matter, worked. The end result is that the build numbers sometimes don't increment for pound-head-against-wall reasons. Good news is that, for once, I'm not doing double duty as the build engineer.) Anyone who thinks he has a better idea of what's good for people than people do is a swine. - P.J. O'Rourke
-
I can only give you my experience. In the past I have done double duty as the build engineer. My current job is the first time this hasn't been the case. The specifics I mention are for my latest company, but reflect how I've also done it in the past. Item 1: All automated builds are available for testing. Testers usually grab the new build, though very rarely they may be in the middle of something and may skip a build. However, they are technically allowed to file bugs only against the latest build (this hasn't been a problem--writing coherent bug reports is a bigger issue.) Once we get to beta testing, all automated builds are available to both internal and external beta testers. We use RTPatch for delivering updates and all builds have RTP update files created for them from the previous build, which gives them the option of doing an update or downloading the entire install (which is quite big.) Item 2: The consensus I've found over the years is that bug fixes should identify the build number in which they were fixed. (The engineer would specific a date and the next build number for benefit of the testers who would then state in the report the earliest build in which they verified the fix.) This requires that versioning be done in a predictable manner. At my current job, we do a "major" build every Tuesday morning, where we increment the build number by 10. For interim builds (there were two this week) the build number is incremented by one. (We only specify the next major build number for fixes.) With every automated build, a notification is sent out containing all the major fixes and whether they are complete or only partially complete, and other readme type things (like; delete all data files and reimport your data.)(Note: To prevent testers from thinking us complete idiots keep the list short, we don't list all the fixes.) (Small confession; the automated build scripts were written years ago by a developer who not only loved perl, but loved to make his code as convoluted as possible and, to add insult to injury, was clueless about how Visual Studio, or even batch files for that matter, worked. The end result is that the build numbers sometimes don't increment for pound-head-against-wall reasons. Good news is that, for once, I'm not doing double duty as the build engineer.) Anyone who thinks he has a better idea of what's good for people than people do is a swine. - P.J. O'Rourke
-
My team is establishing an automated build environment. None of us has any real world experience working with such an environment. We're running into questions about once a build is created, how should test releases, testing, bug tracking, bug fixes, and production releases be "handled"? What I mean by handled is how the process in which these activities are performed and the data that is collected should all be organized. Some example questions: 1) Should each automated build be released for testing, or only selected builds? 2) Does a developer need to indicate in each build which bugs are fixed, or only in a release that is tested? ...and so on. I'm expecting the answers to these questions will all begin with "It depends...", but I'd like to find some resources that will help us make the best decision for how our team does work. Articles, books, or personal experience is welcomed.
Mike Hodnick wrote: 1) Should each automated build be released for testing, or only selected builds? Here, testers fire builds, not developers. I think this answers your question :) Mike Hodnick wrote: 2) Does a developer need to indicate in each build which bugs are fixed, or only in a release that is tested? Actually, we use CVS. On each commit (check-in in other VCS), we put a comment like this: [BugID: 845]. The BugID is a Bugzilla bug#. On each build, a cvs2cl[^] is run, which generates automatically a XML file containing all the fixes (and partial fixes) on this specific version. We process this XML file with a XSL and generate a HTML file containing a user-friendly description and links to our Bugzilla installation for more details. This way, testers have a starting view for deciding what needs re-testing. Due to technical difficulties my previous signature, "I see dumb people" will be off until further notice. Too many people were thinking I was talking about them... :sigh:
-
How does the tester get back to you? Is there a form he fills out or just send over an email saying he I found this bug and that bug, etc.....? There are 10 kinds of people in this world. Those who understand binary and those who don't.
Through a horrible, but functioning, defect tracking system called Alexsys Team. Anyone who thinks he has a better idea of what's good for people than people do is a swine. - P.J. O'Rourke
-
My team is establishing an automated build environment. None of us has any real world experience working with such an environment. We're running into questions about once a build is created, how should test releases, testing, bug tracking, bug fixes, and production releases be "handled"? What I mean by handled is how the process in which these activities are performed and the data that is collected should all be organized. Some example questions: 1) Should each automated build be released for testing, or only selected builds? 2) Does a developer need to indicate in each build which bugs are fixed, or only in a release that is tested? ...and so on. I'm expecting the answers to these questions will all begin with "It depends...", but I'd like to find some resources that will help us make the best decision for how our team does work. Articles, books, or personal experience is welcomed.
Mike Hodnick wrote: I'm expecting the answers to these questions will all begin with "It depends...", but I'd like to find some resources that will help us make the best decision for how our team does work. Articles, books, or personal experience is welcomed. Start simple. Start small. Each step requires the buy-in of the developers and management, because just maintaining an automated build system can itself be a full time job. You may need to use make files instead of a nice solution explorer. Look into NAnt. I'm hosting my own CVS server, which is really easy to do. With the right tools, you can automate the build, the unit tests you run on that build, and error reporting, and it can all be done via the Internet so you can all work at home. It's easy to get lost in the mechanics of setting up an automated build environment, then managing all the information it produces. Ask yourself: why do I need this? what are the top three problems we are having that it needs to solve? Stay focused. :-D Marc Microsoft MVP, Visual C# MyXaml MyXaml Blog
-
Mike Hodnick wrote: I'm expecting the answers to these questions will all begin with "It depends...", but I'd like to find some resources that will help us make the best decision for how our team does work. Articles, books, or personal experience is welcomed. Start simple. Start small. Each step requires the buy-in of the developers and management, because just maintaining an automated build system can itself be a full time job. You may need to use make files instead of a nice solution explorer. Look into NAnt. I'm hosting my own CVS server, which is really easy to do. With the right tools, you can automate the build, the unit tests you run on that build, and error reporting, and it can all be done via the Internet so you can all work at home. It's easy to get lost in the mechanics of setting up an automated build environment, then managing all the information it produces. Ask yourself: why do I need this? what are the top three problems we are having that it needs to solve? Stay focused. :-D Marc Microsoft MVP, Visual C# MyXaml MyXaml Blog
As my development team discusses what we're trying to accomplish, more and more and more ideas keep coming to the table. I appreciate the suggestion to keep focused, as I can see how easily this effort could become a train wreck with ideas and input since we're starting fresh. - Mike ------------------------- "No human being would stack books like that." - Dr. Venkman
-
As my development team discusses what we're trying to accomplish, more and more and more ideas keep coming to the table. I appreciate the suggestion to keep focused, as I can see how easily this effort could become a train wreck with ideas and input since we're starting fresh. - Mike ------------------------- "No human being would stack books like that." - Dr. Venkman
Mike Hodnick wrote: I can see how easily this effort could become a train wreck with ideas and input since we're starting fresh. Exactly. :-D Marc Microsoft MVP, Visual C# MyXaml MyXaml Blog
-
As my development team discusses what we're trying to accomplish, more and more and more ideas keep coming to the table. I appreciate the suggestion to keep focused, as I can see how easily this effort could become a train wreck with ideas and input since we're starting fresh. - Mike ------------------------- "No human being would stack books like that." - Dr. Venkman
Marc has a good point that full automation may not be worthwhile. At my last job, I never fully automated the build since I couldn't justify the time cost. I had to manually change the master version number and start the actually build. After it was done, I quickly checked that there were no errors. If not, I ran a small utility to copy the build out to the release area. I then wrote a short "what's new" email and notified engineering and testing. The whole process took 45 minutes, but only about five of it required my full attention. (At my current company, the fully automated build takes about 15 minutes and we've easily spent much more engineering time fixing the horrible perl scripts than if the install/build engineer just kicked off each step manually, including changing the version number.) Anyone who thinks he has a better idea of what's good for people than people do is a swine. - P.J. O'Rourke
-
My team is establishing an automated build environment. None of us has any real world experience working with such an environment. We're running into questions about once a build is created, how should test releases, testing, bug tracking, bug fixes, and production releases be "handled"? What I mean by handled is how the process in which these activities are performed and the data that is collected should all be organized. Some example questions: 1) Should each automated build be released for testing, or only selected builds? 2) Does a developer need to indicate in each build which bugs are fixed, or only in a release that is tested? ...and so on. I'm expecting the answers to these questions will all begin with "It depends...", but I'd like to find some resources that will help us make the best decision for how our team does work. Articles, books, or personal experience is welcomed.
Take a look at Sentria by www.clearjump.com It can help you to beta test releases (post mortem) and make sure that all release files including the sources and debug symbols are kept in a consistent set. It supports a per release issue tracking as well and many other things.
-
Mike Hodnick wrote: 1) Should each automated build be released for testing, or only selected builds? Here, testers fire builds, not developers. I think this answers your question :) Mike Hodnick wrote: 2) Does a developer need to indicate in each build which bugs are fixed, or only in a release that is tested? Actually, we use CVS. On each commit (check-in in other VCS), we put a comment like this: [BugID: 845]. The BugID is a Bugzilla bug#. On each build, a cvs2cl[^] is run, which generates automatically a XML file containing all the fixes (and partial fixes) on this specific version. We process this XML file with a XSL and generate a HTML file containing a user-friendly description and links to our Bugzilla installation for more details. This way, testers have a starting view for deciding what needs re-testing. Due to technical difficulties my previous signature, "I see dumb people" will be off until further notice. Too many people were thinking I was talking about them... :sigh:
Daniel, I like the way you hook the bug numbers right into the cvs and back out into the docs, simple and functional. :cool: Thanks for the idea!
Stephen Owens