How do you like to be reviewed?
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
Hmmm...If you are a former developer, this should be very easy, right? Huh! ;P Well, you start right with yourself. When you were a developer how did you like to be reviewed? apply the same standards to your developers now. No matter how much you come up with review document policy, at the end of the end of the day, it will be subjective. There is that human factor that will go into it. I guess there is no enough time this time around for that document For next time around, here is what I do in my reviews. - I ask clear questions as to what are my goals and expectations from my boss. - I ask what is the minimum he expects from me. - I ask what do I need to do to get the next promotion, raise, what have you. I write them down and reiterate with my boss what I wrote. Come next review I gauge myself based on what we agreed up on. I have seen this work time and again. With this method, the factor becomes my performance against my goals and less of human factor. I have seen this work for a boss I hated and did not have great relationship. Albeit, we worked together and at the end of a year got both a promotion and a raise ( the only one to get both from the whole team ) ;) So, my best advice is not for this review by for the next one. I think if you clearly communicate with your developers for the next review, there will be less tension come review date.
/* I can C */ // or !C Yusuf
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
I think I might ask myself which IT staff are "indispensable" and why? Then go from there in review, to retain those people. On the other hand... I have been told my "project managers" that indispensable staff pose the biggest risk to any project and need to be "mitigated" for the sake of the project.
MrPlankton
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
Whatever format you come up with, I think it is important to make the review process two-way - i.e., a review of how the manager did as well as the employee. The goals you set should also be two-way - example: the manager agrees to send you to a technical training class. The whole approach should be "How did we do the past year?" rather than "What did you do for me the past year?". Emphasize positives - instead of saying "Stop antagonizing the Marketing Department", say "How do you think you can have a better relationship with Marketing?" or "What can I do to help you have a better relationship with Marketing?" If you feel like you have to be negative in a review, you might as well just fire the person, since a negative review very rarely has a positive outcome.
Best wishes, Hans
[CodeProject Forum Guidelines] [How To Ask A Question] [My Articles]
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
When it comes to things like this I remember a court case I sat in on and had the swaying stance (hung jury until I spoke up).... It wasn't about programming, rather teaching, and turning someone down for tenure. Ignoring the actual job, but focusing on the meat of the subject. Everyone has expectations, you can say all you want raises should not be expected, but be honest, they always will be. So make it clear what is expected to get a good review, the categories, the weights, and what "extra" is expected to be considered "going the extra mile" for the company. What is expected and what is considered above that, and how? In that particular case the teacher was denied tenure because she did not finish her training in the time alotted. Her complaint was that her boss did not put anything as a requirement until the last year... and then she couldn't fit all the classes in on that year so should not be held responsible for not getting the training.... My comment in the back room was what we do as employees when we see a comment from our boss. for six years there was in the "suggestions" column "you need to catch up on training in ...." On the seventh, it moved to the requirements column, and then the employee did something. Sure my boss may put something in the suggestion column on mine, and I may gloss over it the first time, but if it shows up again... hey wait a second! That is when I started asking, how important is this? The answer is there are always two divisions of performance, that which is needed to "just do the job" and that which is needed to do "an exceptional job" and there are all the states in between, but generally you judge the value between those two positions.... There is the third "not doing the job" and judging a value betwen that and "just doing the job" determines how long you keep them, or if you do. I don't believe the employer should have to tell the employee every year you need to do this, but I like the suggestions of goals, what are the minimum, what is considered a level of extra effort worthy of a raise. Is there any areas in the company that are lacking skills that I have that I could help out in? Interest in the company is great for the employee, interest in the employee is great for the company. just my 2 cents worth.... keep the change... ;)
_________________________ Asu no koto o ieba, tenjo de nezumi ga warau. Talk about things of tomorrow and the mice in the ceiling laugh. (Japanese Proverb)
-
Hmmm...If you are a former developer, this should be very easy, right? Huh! ;P Well, you start right with yourself. When you were a developer how did you like to be reviewed? apply the same standards to your developers now. No matter how much you come up with review document policy, at the end of the end of the day, it will be subjective. There is that human factor that will go into it. I guess there is no enough time this time around for that document For next time around, here is what I do in my reviews. - I ask clear questions as to what are my goals and expectations from my boss. - I ask what is the minimum he expects from me. - I ask what do I need to do to get the next promotion, raise, what have you. I write them down and reiterate with my boss what I wrote. Come next review I gauge myself based on what we agreed up on. I have seen this work time and again. With this method, the factor becomes my performance against my goals and less of human factor. I have seen this work for a boss I hated and did not have great relationship. Albeit, we worked together and at the end of a year got both a promotion and a raise ( the only one to get both from the whole team ) ;) So, my best advice is not for this review by for the next one. I think if you clearly communicate with your developers for the next review, there will be less tension come review date.
/* I can C */ // or !C Yusuf
Well, that's the thing. In each of my jobs in the last 10 years, I was either the only developer, or I was only a developer for a short period of time before I got promoted to lead or management. I have never really had a "developer" review. I was always reviewed on how I got along with others, whether I was late for work or excessive absences, etc. My code or output was never put into question because there was nobody qualified to review that. They figured the product worked, and it had new features, I must be doing my job. So as much as I would like to come from my own experience, I have none in this area. Dan
-
I think I might ask myself which IT staff are "indispensable" and why? Then go from there in review, to retain those people. On the other hand... I have been told my "project managers" that indispensable staff pose the biggest risk to any project and need to be "mitigated" for the sake of the project.
MrPlankton
MrPlankton wrote:
I think I might ask myself which IT staff are "indispensable" and why? Then go from there in review, to retain those people. On the other hand... I have been told my "project managers" that indispensable staff pose the biggest risk to any project and need to be "mitigated" for the sake of the project.
Both are true. People are indispensible because they combine (a) excellent skills and (b) domain knowledge. If I felt arsed I'd find the links to two papers (both Havard I believe) that calculated the direct and indirect cost of losing skilled employees at over 1yrs salary. However one should also apply the bus question, if "x" got hit by a bus tomorrow would we be screwed? If the answer is yes then it's time to pass domain knowledge on or hire another genius to share the responsibility (not as some advocate, fire the most productive member of your team - that's a last resort if they are both indispensable and intransigent about spreading the risk - and then only after enough domain knowledge has been extracted through whatever means). Fixed typos
I'm largely language agnostic
After a while they all bug me :doh:
modified on Thursday, February 07, 2008 12:52:51 PM
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
Take a few tasks assigned and then take the time taken to complete the task and salary compared to peers. Use your expertise in judging how efficiently the task was completed as well as how correct. (Commented, best practices, innovative problem solving, bugs introduced, etc.) Also, direct comparison to peers can help. It establishes the mentoring chain. As well as an expected level of success.
Need a C# Consultant? I'm available.
Happiness in intelligent people is the rarest thing I know. -- Ernest Hemingway -
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
So you have never, in all your other management experiences, given out reviews? You say you have never had your own code reviewed in a meaningful manner by anyone with the skills to criticise your code. You do mention a talent for getting placed in supervisory positions before much of your work reaches production. Now you have been appointed CIO. But you still need to 'get permission' to make changes in your internal review process. My suggestion would to find someone who is generally respected by the development staff and turn the entire process over to him/her.
Jon Smith & Wesson: The original point and click interface
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
Well, you're the one in management. It's up to you, not what other people may suggest on a message board or what you read in a book or retain from a seminar. Management consists of managing "the company you represent" and it's resources (that make up the company). What may work for a Fortune 100 company with 500 developers will not work for a 3 person LLC and vice versa. That's the art of management: working with what you have within constraints. What you need to do is identify your boundaries and make a decision (good or bad) based on how you view the business, staff and how you can deal with any fall-out (if it backfires) or success (if it works) I don't mean to come across as harsh or anything. Putting it this way, it's like asking for marriage advice from stangers that don't know you or your relationship. It's the same in business (unfortunately). Not knowing the intricacies of your company, it's hard to give any advice other than it's your own call. I know this doesn't answer your question one bit but maybe it'll help you in coming up with your own methodology.
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
There are only two questions I care about, whether the developer or the manager: How have I failed to support you in getting your job done? How have you failed to support me in getting my job done? That's it. From there, with honesty, the sky is the limit to really getting down to the issues that are at hand. And yes, I tend to live by the motto "success is not an option". Marc
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
If the only problem you can find right now is to much "churn" then consider yourself lucky. I would try to address the "churn" issue department wide (from design/spec to code to qa).
Todd Smith
-
Well, you're the one in management. It's up to you, not what other people may suggest on a message board or what you read in a book or retain from a seminar. Management consists of managing "the company you represent" and it's resources (that make up the company). What may work for a Fortune 100 company with 500 developers will not work for a 3 person LLC and vice versa. That's the art of management: working with what you have within constraints. What you need to do is identify your boundaries and make a decision (good or bad) based on how you view the business, staff and how you can deal with any fall-out (if it backfires) or success (if it works) I don't mean to come across as harsh or anything. Putting it this way, it's like asking for marriage advice from stangers that don't know you or your relationship. It's the same in business (unfortunately). Not knowing the intricacies of your company, it's hard to give any advice other than it's your own call. I know this doesn't answer your question one bit but maybe it'll help you in coming up with your own methodology.
Actually, I disagree with your comment. I'm not asking for help in managing developers. I can manage them just fine, otherwise I wouldn't be in this position. It wasn't created just for me. Since I do not have experience in being reviewed as a developer, I was asking for guidance on how other developers get reviewed at their companies, and whether they feel that's a good approach. I don't think it should really matter whether you have 5 or 500 developers, because performance can still be measured the same way. You have the same potential for code churn, you still have the team relationship aspect unless you're the only coder, etc. The components of how your performance is reviewed should be relatively constant across companies of different sizes. My post was concerning ways I could better review my employees rather than grading them 1 through 5 on how often they are tardy, or 1 through 5 on how they interact with customers and so on. A developer's performance isn't tied to the business except by management. If you're not meeting the goals of the business, that's because your manager didn't properly map those goals into work items for your position. On the other hand, if I placed goals directly on you that I felt contributed to the overall business goal, and you fell short, now I have something to review you on. You didn't fail the business goal, you failed my goal for you. I failed the business goal as a result of you not meeting your goals, and that will be addressed in my review with the CEO. The developer does the project laid out before them, and if they completed those projects in the specified time, they probably met their goals and as a result, I met the goals placed on me by my boss. I'm simply looking for ideas on reviewing developers that don't involve ranking them 1 through 5 on their language and grammar. I want a more productive approach than "What are the employee's shortcomings and weaknesses?". Nobody in this forum is going to create "my mthodology" for me, but the insight I receive from the developers on this site might certainly highlight methods to avoid, try or embrace.
-
If the only problem you can find right now is to much "churn" then consider yourself lucky. I would try to address the "churn" issue department wide (from design/spec to code to qa).
Todd Smith
-
I am a former C# developer, turned CIO for a commercial software company. I say former developer because even though I open VS2005 every day with the hope of writing some code, it never seems to happen with everything else going on. I am now faced with doing performance reviews on the people who I used to code beside. The "official company document" for reviewing employees seems completely useless when it comes to developers. I have been given permission to take any approach necessary to create a more appropriate review document, so I am asking for input! What should I be reviewing the development staff on? One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for. We're going to be getting into TDD, but the review topic is still valid. How do you expect to be reviewed in a development shop where you are one of numerous developers, DBA's, architects and testers? Dan
AlaskaDan wrote:
One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for.
Well if you are looking for shortcuts to quality I don't believe there are any. Try looking into Software Development Best Practices and Principles. You could try reading people like Kent Beck, Martin Fowler, Ward Cunningham, Alan Holub, etc. There's a quite large list of leaders in the industry these days. They don't all agree on every point but if you could implement just 10% of the 80% they do agree on you might improve your situation like 1000%. Or I have no idea what I am talking about.
led mike
-
Actually, I disagree with your comment. I'm not asking for help in managing developers. I can manage them just fine, otherwise I wouldn't be in this position. It wasn't created just for me. Since I do not have experience in being reviewed as a developer, I was asking for guidance on how other developers get reviewed at their companies, and whether they feel that's a good approach. I don't think it should really matter whether you have 5 or 500 developers, because performance can still be measured the same way. You have the same potential for code churn, you still have the team relationship aspect unless you're the only coder, etc. The components of how your performance is reviewed should be relatively constant across companies of different sizes. My post was concerning ways I could better review my employees rather than grading them 1 through 5 on how often they are tardy, or 1 through 5 on how they interact with customers and so on. A developer's performance isn't tied to the business except by management. If you're not meeting the goals of the business, that's because your manager didn't properly map those goals into work items for your position. On the other hand, if I placed goals directly on you that I felt contributed to the overall business goal, and you fell short, now I have something to review you on. You didn't fail the business goal, you failed my goal for you. I failed the business goal as a result of you not meeting your goals, and that will be addressed in my review with the CEO. The developer does the project laid out before them, and if they completed those projects in the specified time, they probably met their goals and as a result, I met the goals placed on me by my boss. I'm simply looking for ideas on reviewing developers that don't involve ranking them 1 through 5 on their language and grammar. I want a more productive approach than "What are the employee's shortcomings and weaknesses?". Nobody in this forum is going to create "my mthodology" for me, but the insight I receive from the developers on this site might certainly highlight methods to avoid, try or embrace.
I wasn't giving you directions on "managing developers". My response was that only you know how your business is run, what the job market is like, what concentration these developers offer to your end success and such. My point is that it's more or less a case by case basis rather than generalities. What works as a review for one company will/may not work for another. For example: Medical imbeded software: Mission critical, stability is key, deadlines aren't the emphasis. Financial services software: Time to market is critical, deadlines are. So, I wouldn't use the same methodology for Medical as I would for Financial Services. The focus of one is different from the other so I would treat reviews based on the company focus. Certain requirements for one may be vastly different than the other. What I'm saying is that you know the business so it's really up to you to figure out what's important to the company and how it operates and base your decision on that. Developers may be considered plain vanilla but I think how they integrate into the organization and it's goals is what's most important. That's just my opinion. That's what I would base my reviews on.
-
Actually, I disagree with your comment. I'm not asking for help in managing developers. I can manage them just fine, otherwise I wouldn't be in this position. It wasn't created just for me. Since I do not have experience in being reviewed as a developer, I was asking for guidance on how other developers get reviewed at their companies, and whether they feel that's a good approach. I don't think it should really matter whether you have 5 or 500 developers, because performance can still be measured the same way. You have the same potential for code churn, you still have the team relationship aspect unless you're the only coder, etc. The components of how your performance is reviewed should be relatively constant across companies of different sizes. My post was concerning ways I could better review my employees rather than grading them 1 through 5 on how often they are tardy, or 1 through 5 on how they interact with customers and so on. A developer's performance isn't tied to the business except by management. If you're not meeting the goals of the business, that's because your manager didn't properly map those goals into work items for your position. On the other hand, if I placed goals directly on you that I felt contributed to the overall business goal, and you fell short, now I have something to review you on. You didn't fail the business goal, you failed my goal for you. I failed the business goal as a result of you not meeting your goals, and that will be addressed in my review with the CEO. The developer does the project laid out before them, and if they completed those projects in the specified time, they probably met their goals and as a result, I met the goals placed on me by my boss. I'm simply looking for ideas on reviewing developers that don't involve ranking them 1 through 5 on their language and grammar. I want a more productive approach than "What are the employee's shortcomings and weaknesses?". Nobody in this forum is going to create "my mthodology" for me, but the insight I receive from the developers on this site might certainly highlight methods to avoid, try or embrace.
"On the other hand, if I placed goals directly on you that I felt contributed to the overall business goal, and you fell short, now I have something to review you on. You didn't fail the business goal, you failed my goal for you. I failed the business goal as a result of you not meeting your goals, and that will be addressed in my review with the CEO. The developer does the project laid out before them, and if they completed those projects in the specified time, they probably met their goals and as a result, I met the goals placed on me by my boss." Well said, and that paragraph is your answer to figuring out reviews. Just read what you wrote and adapt that into a review method.
-
So you have never, in all your other management experiences, given out reviews? You say you have never had your own code reviewed in a meaningful manner by anyone with the skills to criticise your code. You do mention a talent for getting placed in supervisory positions before much of your work reaches production. Now you have been appointed CIO. But you still need to 'get permission' to make changes in your internal review process. My suggestion would to find someone who is generally respected by the development staff and turn the entire process over to him/her.
Jon Smith & Wesson: The original point and click interface
I have done reviews, but they were all the canned reviews that the entire company uses. I just feel like a developer should be reviewed differently than the secretary. The tech support people should be reviewed differently from the QA people, and so on. Each position has totally different requirements and qualities. One template the company downloaded 8 years ago when there were only 5 employees is outdated now that we have 30 employees and the developer isn't some guy who was writing Access in his mom's basement while he worked at Wendy's for his day job. The only reason I needed permission was because HR was giving me grief about not wanting to use the template, so I went to the CEO so it could be "official". Also, I never said I get promoted before my work reaches production. All of my work has reached production, I was just trying to keep up my skills by retaining 1 project that I had when I was part of the development team. My previous reviews have been conducted with the canned review templates and didn't address my coding ability at all. That's what I want to change with my new position. The current template doesn't address the core piece of a developer's job, the code! Dan
-
"On the other hand, if I placed goals directly on you that I felt contributed to the overall business goal, and you fell short, now I have something to review you on. You didn't fail the business goal, you failed my goal for you. I failed the business goal as a result of you not meeting your goals, and that will be addressed in my review with the CEO. The developer does the project laid out before them, and if they completed those projects in the specified time, they probably met their goals and as a result, I met the goals placed on me by my boss." Well said, and that paragraph is your answer to figuring out reviews. Just read what you wrote and adapt that into a review method.
-
AlaskaDan wrote:
One of my issues is code churn. There is too much back and forth between development and QA for something the developer could have easily tested for.
Well if you are looking for shortcuts to quality I don't believe there are any. Try looking into Software Development Best Practices and Principles. You could try reading people like Kent Beck, Martin Fowler, Ward Cunningham, Alan Holub, etc. There's a quite large list of leaders in the industry these days. They don't all agree on every point but if you could implement just 10% of the 80% they do agree on you might improve your situation like 1000%. Or I have no idea what I am talking about.
led mike
I'm not looking for shortcuts to quality at all. On the contrary, I'm looking to improve quality 10 fold. Before I got promoted, you gave the developer a basic description of the feature to be developed, and sent them on their way. There was no real "end case" defined, no description of what would consider the feature complete in the eyes of the stakeholders, and certainly no definition of test cases to consider the item ready for QA. That is all changing now, but it will be an evolving process.